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Abstract 

Air  traffic  controllers  are  responsible  for  directing  air  traffic  based  upon  decisions 
made  from  traffic  activity  depicted  on  2Dimensional  (2D)  radar  displays.  Controllers 
must  identify  aircraft  and  detect  potential  conflicts  while  simultaneously  developing  and 
executing  plans  of  action  to  ensure  safe  separation  is  maintained.  With  a  nearly  100% 
increase  in  traffic  expected  within  the  next  decade  (FAA,  2012a),  controllers’  abilities  to 
rapidly  interpret  spacing  and  maintain  awareness  for  longer  durations  with  increased 
workload  will  become  increasingly  imperative  to  safety. 

The  current  display  design  spatially  depicts  an  aircraft’s  position  relative  to  the 
controller’s  airspace  as  well  as  speed,  altitude,  and  direction  in  textual  form  which 
requires  deciphering  and  arithmetic  to  determine  vertical  separation.  Since  vertical 
separation  is  as  imperative  to  flight  safety  as  lateral  separation,  affording  the  controller  an 
intuitive  design  for  determining  spacing  without  mental  model  creation  is  critical  to 
reducing  controller  workload,  and  increasing  awareness  and  efficiency.  To  examine  this 
potential,  a  stereoscopic  radar  workstation  simulator  was  developed  and  field-tested  with 
35  USAF  controllers.  It  presented  a  view  similar  to  traditional  radar  displays,  (i.e.  top- 
down),  however,  it  depicted  altitude  through  the  use  of  3D  stereoscopic  disparity, 
permitting  vertical  separation  to  be  visually  represented. 
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Effects  of  Stereoscopic  3D  Digital  Radar  Displays  on  Air  Traffic  Controller 

Performance 

1.  Introduction 

“Our  Airmen's  ability  to  rethink  the  battle  while  incorporating  new  technologies  will 
improve  the  varied  ways  our  Air  Force  accomplishes  its  mission.” 

-  Gen.  Welch,  excerpt  from  “A  Vision  for  the  United  States  Air  Force”,  2012 

General  Issue 

Air  traffic  control  (ATC)  in  the  US  is  in  need  of  a  comprehensive  overhaul  to 
maintain  the  present  standards  of  safety  and  efficiency  as  the  national  airspace  system 
(NAS)  is  pushed  to  capacity  (Marsa,  2009).  This  need  has  resulted  in  the  Federal 
Aviation  Administrations’  (FAA)  ongoing  initiative  named  Next  Generation  Air  traffic 
Control  or  “NextGen”.  In  an  effort  to  increase  the  capabilities  of  the  existing  NAS  and 
supplement  the  controller  workforce,  there  will  be  a  heavy  focus  on  automating  many 
control  functions  as  new  technologies  mature.  This  change  is  anticipated  to  redefine 
controllers’  roles  from  that  of  active  controllers  to  passive  automation  monitors. 

Although  NextGen  technologies  are  years  from  replacing  human  controllers, 
their  implications  highlighted  the  importance  of  understanding  vigilance  limits  (Wickens, 
1998).  Projected  air  traffic  increases  combined  with  budget  cuts  hinder  the  ability  to  hire 
more  controllers,  thus  lengthening  controller  work  durations  to  cope  with  demand.  These 
changes  make  human  factors  (HF)  limitations  increasingly  critical  to  understand  in  the 
context  of  air  traffic  controllers  (Erzberger,  2004).  This  is  especially  true  when  one 
considers  controllers’  primary  function  is  to  maintain  a  heightened  level  of  vigilance  and 
situation  awareness  (SA)  to  detect  deviations  and  separation  losses  to  predict  and  avoid 
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potential  conflicts.  This  workload  is  mentally  tasking  and  understanding  the  ways  to 
manage,  measure,  and  mitigate  the  negative  impact  of  this  workload  is  crucial. 

Current  ATC  in  a  radar  environment  requires  constant  mental  calculations  to  infer 
the  vertical  distance  between  aircraft  by  adding  and  subtracting  the  textual  display  of 
altitude  for  many  aircraft.  Additionally,  this  task  requires  the  projection  of  trajectory  and 
altitude  changes.  Each  of  these  are  performed  often  dozens  of  times  a  minute.  These 
mental  manipulations  allow  a  controller  to  create  a  “mental  3D  image”  of  the  traffic 
scene  through  mental  interpretation  of  alphanumerical  altitude  data  (Wong  et  al.,  2008). 
This  is  known  to  be  a  mentally  fatiguing  practice  and  current  stereoscopic  display 
technology  exists  that  may  permit  these  mental  calculations  to  be  reduced.  Due  to  the 
rapid  pace  and  complex  nature  of  ATC  duty  and  the  HF  limitations  regarding  vigilance, 
SA  and  fatigue,  it  is  imperative  that  any  technology  with  the  potential  to  increase  human 
performance  in  these  areas  be  explored. 

These  particular  aspects  of  ATC  HF  have  recently  been  highlighted  with  several 
highly  publicized  air  traffic  controller  deviations.  In  Kentucky  a  severely  fatigued 
controller,  having  gotten  a  mere  2  hours  of  broken  sleep  in  the  24  hours  leading  up  to  the 
incident,  cleared  an  aircraft  for  takeoff  on  the  wrong  runway  leading  to  a  mishap 
including  49  fatalities  (NTSB,  2007).  Additional  suspicion  can  be  raised  for  the  human 
performance  implications  evidenced  by  a  rash  of  several  instances  where  controllers  were 
caught  falling  asleep  on  watch  among  other  infractions  nationwide  (Funa,  1997). 
Additional  errors  of  perception  or  task  saturation  through  workload  have  led  to  notable 
failures  and  mishaps  in  the  system.  Flight  safety  is  critical  for  the  future  success  of  our  air 
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transportation  system  and  the  HF  aspects  of  ATC  must  be  fully  understood  to  mitigate 
human  error  potential  to  the  maximum  extent. 

Tasked  with  managing  the  safe  and  expeditious  flow  of  aircraft;  separating  them 
from  one  another  and  terrain;  it  is  incumbent  upon  the  controllers  to  remain  engaged  and 
vigilant,  detecting  and  correcting  deviations  and  conflicts  (FAA,  2010).  Air  traffic 
controllers  are  responsible  for  directing  air  traffic  based  upon  decisions  made  using  vast 
amounts  of  ever-changing  information  from  a  diverse  array  of  peripheral  (static  and 
dynamic)  displays  and  presentations,  (FAA,  2012:2-1-1).  Their  task  includes  identifying 
aircraft  and  detecting  potential  conflicts  among  them,  while  simultaneously  developing 
and  executing  plans  of  action  for  all  aircraft  within  the  entire  airspace.  With  the 
exponential  increase  in  air  traffic  expected  to  continue  in  to  the  foreseeable  future, 
controllers’  abilities  to  maintain  vigilance  and  heightened  situation  awareness  for  longer 
durations  with  increased  workload  will  become  even  more  imperative  to  safe  ATC 
operations  within  the  National  Airspace  System  (NAS). 

In  the  FAA's  NextGen  scheme,  Automatic  Dependent  Surveillance-Broadcast 

System  (ADS-B)  equipped  aircraft  will  revolutionize  how  aircraft  are  separated 

(McCallie,  2011;  Magazu  III,  2012).  This  new  technology  incorporated  under  NextGen 

will  permit  aircraft  to  continuously  broadcast  aircraft  location,  speed,  and  flight 

information  and  provide  this  information  in  each  aircraft  to  create  automatically  self- 

separating  traffic  (Me  Callie,  2011).  Therefore  controllers  are  expected  to  transition  to 

passive  roles  of  monitoring  self-separating  traffic  and  become  responsible  for  monitoring 

larger  numbers  of  aircraft.  As  a  result,  the  workload,  tasks  and  workstations  of  these 

controllers  will  change  dramatically  (Prevot,  Hemola  and  Mercer,  2008).  Of  particular 
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interest  is  the  propensity  for  humans  to  become  complacent  in  situations  where  active 
involvement  and  mental  engagement  is  limited  or  omitted  resulting  in  waning  awareness 
and  vigilance  decrement. 

As  controllers  assume  this  monitoring  role,  it  also  stands  to  reason  that  the  area  of 
responsibility  for  each  controller  will  increase— as  well  as  the  volume  of  traffic  in  their 
given  sector  of  responsibility— as  aircraft  self-separate,  the  equivalent  perceived  workload 
will  be  reduced  hence  the  increase  in  aircraft  to  controller  ratio.  This  may  prove  to  be  a 
dangerous  transition  from  a  HF  standpoint  unless  the  human  performance  limitations  are 
clearly  understood  and  addressed  by  providing  operators  with  the  best  presentations 
possible  to  execute  their  duties  intuitively.  At  a  minimum,  the  displays  used  for  this  task 
ought  to  be  as  advanced  as  the  system  in  place  for  self- separation  and  permit  a  controller 
to  see  a  comprehensive  representation,  aerial  "airspace  activity  model"  commensurate  to 
the  level  of  complexity  being  experienced  and  shown  in  an  easy  to  comprehend  manner. 

A  particularly  important  display  is  the  radar  workstation,  which  portrays  a  God's 
eye  view  of  the  airspace,  showing  a  2D,  top  down  representation  of  each  aircraft. 
Supplementary  information,  including  aircraft  identification  and  altitude,  are  displayed 
through  text  labels  assigned  to  each  aircraft,  often  referred  to  as  "data  blocks".  When 
using  this  workstation  the  air  traffic  controller  monitors  the  location  of  each  aircraft 
within  the  three-dimensional  airspace,  creates  an  understanding  of  the  location  of  each 
aircraft  by  viewing  their  location  against  a  ground  plane  and  then  reading  and  temporarily 
memorizing  the  altitudes  of  each  aircraft  to  complete  their  mental  representation.  Below, 
Figure  1  is  an  example  of  a  current  USAF  digital  radar  display  workstation. 
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Figure  1:  USAF  Controllers  at  Present  Day  Digital  Radar  Workstation 


In  an  effort  to  improve  upon  the  current  ATC  display,  a  simulated  stereoscopic 
radar  workstation  was  developed  and  field-tested  with  35  USAF  controllers.  This 
workstation  presents  a  view  similar  to  traditional  radar  displays  but  depicts  altitude 
through  the  use  of  stereoscopic  disparity,  permitting  vertical  separation  to  be  visually 
represented  as  differences  in  disparity.  This  top-down  view  is  intended  to  reduce  clutter 
by  omitting,  through  toggle  selection,  supplementary  “data  blocks”  (superfluous  when 
particular  aircraft  are  not  of  primary  or  immediate  concern)  that  hold  altitude  codes, 
replacing  them  with  visual  disparity  cues.  Furthermore,  the  workstation  is  intended  to 
reduce  “scan”  frequency,  (when  a  controller  visually  scans  their  entire  area  of 
responsibility  to  detect  and  identify  potential  conflicts),  minimizing  lost  time  and 
distraction,  or  SA  decrement,  from  unnecessary  scans. 

As  the  final  responsibility  for  flight  safety  remains  with  the  human  controllers, 
their  ability  to  observe  the  increased  traffic  volume  accurately  and  in  an  intuitive  manner 
will  be  increasingly  critical  for  maximizing  efficiency,  maintaining  awareness,  and 
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limiting  fatigue.  Displays  are  all  controllers  have  to  rely  on  for  ensuring  the  safety  of 
millions  of  flights  a  year,  so  their  presentations  should  be  as  easily  understood  as  possible 
while  providing  the  most  accurate  representation  of  real-time  traffic.  See  below  figure 
for  example  of  traffic  increases  as  displayed  on  a  radar  scope. 


Figure  2:  Current  day  ATC  display  with  lx,  2x,  and  3x  traffic  load  (Prevot,  Homola  & 

Mercer,  2008) 

Research  Objectives  and  Hypotheses 

This  research  investigates  and  identifies  some  potential  impacts  of  stereoscopic 
displays  in  ATC  such  as  fatigue  reduction,  decision-making,  perceived  workload  and 
situation  awareness  enhancement.  The  potential  for  a  controller’s  vigilance  to  wane 
rapidly  when  not  actively  engaged  in  communications  and  making  separation  decisions  is 
a  significant  concern  in  air  transportation  safety  and  recognized  as  the  primary  limitation 
to  industry  growth  (Parasuraman,  Molloy  &  Singh,  1993.  There  exists  a  need  to  evaluate 
the  specific  human  factors  involved  to  ensure  human  performance  in  this  capacity  before 
auto-separating  NextGen  ATC  it  is  fully  implemented  (Salas  &  Maurino,  2010). 
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This  paper  will  propose  a  new  stereoscopic  workstation;  including  its  design, 
expected  utility,  field  experiment,  and  an  analysis  of  data  collected  from  the  field 
experiment. 

The  objective  and  goal  of  this  research  is  to  determine  the  potential  positive  and 
negative  impacts  of  stereoscopic  3D  when  used  in  the  complex  application  context  of  air 
traffic  control.  From  a  human  factors  and  performance  perspective,  it  is  imperative  to 
determine  how  this  technology  may  specifically  improve  controller  effectiveness  through 
heightened  situation  awareness  and  reduced  mental  fatigue  and  strain.  Also  of  concern  is 
the  potential  efficiency  improvement  by  permitting  controllers  to  infer  data  more  easily 
through  a  simplistic  and  “natural”  presentation  of  aircraft  altitudes  thus  allowing  one  to 
rapidly  determine  spatial  orientation  of  targets  in  a  simulated  3D  dynamic  model  of  the 
airspace. 

This  technology  and  the  prototype  created  for  this  research  has  specifically  been 
intended  to  be  used  as  a  “decision  aid”.  This  determination  may  seem  like  a  limitation, 
but  the  controller  must  be  afforded  immediate  access  to  rapidly  changing  information 
upon  which  to  make  vital  decisions  in  the  interest  of  flight  safety.  Therefore,  any  decision 
aid  enhancing  the  controller’s  ability  to  gain,  process  and  use  the  information  presented 
in  a  radar  environment  is  therefore  a  benefit. 

Controller  reliance  on  peripheral  automation  and  digitized  radar  displays  has 

grown  and  its  prevalence  in  the  workspace  has  rendered  it  indispensible  for  all  duties 

related  to  ATC.  Therefore  making  the  best  use  of  the  technology  available  to  ensure  a 

controller  has  the  most  accurate,  timely,  and  most  importantly,  easy  to  comprehend 

information  upon  which  to  base  decisions  is  imperative  to  flight  safety  in  every  segment 
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of  controlled  airspace  in  the  NAS.  From  a  HF  perspective,  the  displays  must  not  only  be 
intuitive,  they  must  present  information  and  raw  data  the  controller  needs  quickly,  ready 
to  employ,  and  be  used  to  support  controller's  decision  making  in  a  specifically  pertinent 
manner  conducive  to  the  specialist’s  duties;  in  this  case,  separation  of  aircraft.  Therefore 
the  research  question  of  this  paper  is  therefore: 

Does  stereoscopic  3D  presentation  of  digital  radar  displays  enhance 

controller  performance  and  effectiveness? 

Methodology 

As  previously  indicated,  the  methodology  utilized  in  this  study  was  that  of  an 
experiment  carried  out  on  actual  USAF  air  traffic  controllers.  Chapter  III  contains 
extensive  details  of  the  methods  and  procedures  utilized  in  accomplishing  this. 

Summary 

This  chapter  introduced  the  reasons  for  and  manner  in  which  this  research  was 
approached  and  conducted  as  well  as  some  early  predictions  of  the  impact.  The  next 
chapter,  Chapter  II,  will  detail  what  the  predominant  literature  in  these  areas  indicates. 
Within  this  review,  human  factors,  human-machine-interaction,  display  technology,  and 
performance  measures  along  with  empirical  evidence  from  earlier  works  will  be 
discussed.  Hypotheses  will  be  formed  and  outlined  in  this  chapter  as  well.  Chapter  III 
will  include  an  in-depth  disclosure  of  the  manner  in  which  this  research  experiment  was 
carried  out  to  include  all  aspects  of  the  experimental  design,  employment,  focus  and 
some  preliminary  findings.  Next,  Chapter  IV  will  further  elaborate  upon  the  results 
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gathered  from  the  participants  during  the  experiment.  Using  statistical  analysis,  those 
results  will  be  discussed  and  compared  to  the  hypotheses  laid  out  in  Chapter  II. 
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II.  Literature  Review 


Chapter  Overview 

In  this  chapter,  relevant  research  pertaining  to  the  enhancement,  testing  and 
creation  of  air  traffic  controller  displays  will  be  explored.  It  contains  an  extensive, 
however  by  no  means  exhaustive,  literature  review  with  specific  focuses  upon  the 
implications  of  and  varied  applications  for  many  types  of  3D  display  applications  for 
ATC.  This  chapter  results  in  the  generation  of  six  specific  hypotheses,  a  variables 
relationship  model  and  measurables  for  each  metric.  There  will  also  be  a  brief  discussion 
pertaining  to  the  overarching  requirement  for  high  situation  awareness  that  impacts  the 
entire  discipline  as  well  as  a  theory  that  best  supports  the  need  for  this  technology’s 
development  to  provide  this  capacity. 

ATC  Considerations 

The  NAS  consists  of  a  vast  network  of  people  and  equipment  designed  to  provide 
a  safe  and  efficient  national  aviation  environment.  ATC  is  a  service  provided  by  ground- 
based  controllers  that  work  within  the  NAS  to  direct  aircraft  on  the  ground  and  in  the  air. 
Controllers  are  responsible  for  managing  and  coordinating  the  flow  of  air  traffic  and 
ensuring  safe  flights,  arrivals  and  departures,  while  maximizing  efficiencies  where 
possible  and  ensuring  safety  above  all  through  the  use  of  standard  separation  minima 
between  aircraft  as  prescribed  by  the  FAA. 

ATC  is  a  dynamic  environment  where  controllers  constantly  receive  a  large 
volume  of  information  from  multiple  sources  to  monitor  the  changes  in  the  environment, 
make  decisions,  and  perform  effective  actions  in  a  timely  manner  (Xing  &  Manning, 
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2005).  The  role  of  controllers  is  a  critical  one.  The  ATC  system’s  intricacies  include 
thousands  of  separate  facilities  all  communicating  with  each  other  and  handling 
information  via  different  sources  (e.g.,  radar  screen  with  a  series  of  automated  visual 
cues,  paper  or  electronic  flight  progress  strips,  radio  and  interphone  communication). 

Air  traffic  controllers  have  to  deal  with  all  these  different  sources  of  information  to 
identify  potential  conflicts  and  along  with  strict  regulatory  guidance,  provide  timely 
resolution  of  these  potential  conflicts. 

Background  and  Related  Literature 

An  enormous  amount  of  research  has  been  conducted  to  understand  the 
application  of  stereoscopic  displays  for  command  and  control,  air  traffic,  and  system 
operators  of  every  sort.  Although  stereoscopic  displays  have  been  around  for  over  150 
years,  including  Wheatstone’s  Stereograph  created  in  1838  (Mclntire  et  al,  2012),  the 
current  surge  of  interest  is  attributable  in  part  to  the  vastly  improved  quality  of  the 
display  methods  currently  in  use.  Additionally,  societal  impacts  such  as  3D  use  in 
mediums  like  video  games,  movies,  home  theaters  and  televisions,  even  mobile  phones 
and  tablets  has  led  to  more  acceptance  of  the  technology  in  mainstream  culture. 

Stereoscopic  3D  is  a  method  of  viewing  2D  images  or  dynamic  scenes  by  tricking 
the  brain  into  detecting  depth.  Human  vision  is  already  in  stereo  as  each  eye  is 
positioned  differently  in  the  skull  providing  a  slightly  different  perspective.  By  providing 
each  eye  with  images  from  a  slightly  varied  perspective,  or  angle  of  view,  the  images 
seen  separately  by  each  eye  will  be  fused  into  singular  images  in  the  brain  to  gain  a  sense 
of  depth,  even  when  each  eye  is  only  seeing  a  2D  image.  Active  shutter  stereoscopic 
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technology  allows  clarity  by  eliminating  “crosstalk”,  a  phenomenon  that  occurs  when 
eyes  see  images  intended  for  the  other.  Active  shutter  actually  blocks  the  non-viewing 
eye  while  the  other  views  and  vice  versa  so  images  only  reach  the  intended  eye — this 
equates  to  high  fidelity  3D  views  in  most  cases.  Stereoscopic  viewing  can  induce  eye- 
strain,  fatigue,  and  other  physiological  discomfort  such  as  feelings  of  disorientation  or 
nausea  if  improperly  set-up.  With  responsible  configuration  and  management  of  the 
equipment,  comfortable  long-term  viewing  can  be  achieved.  Some  other  types  of  3D 
viewing  exist  and  have  been  explored  for  ATC  and  will  be  discussed  later  in  this  paper. 

Information  is  abundant,  however,  having  what  you  need  when  you  need  it  is  no 
longer  the  problem;  it  is  the  ability  to  access  and  process  it  for  use  quickly  that  has 
become  the  limiting  factor.  This  requirement  falls  squarely  on  the  information 
technology  and  human-computer-interaction  fields  as  the  need  for  better  organization  of 
displayed  information  along  with  intuitive  displays  is  demanded.  With  the  sheer  amount 
of  available  information  we  attempt  to  present  in  any  given  new  display,  the  manner  in 
which  it  is  presented  may  be  instrumental  in  its  usability,  making  stereoscopic  views  a 
potential  enhancement  for  complex  information  systems. 

Human  Factors  and  ATC  Performance 

Air  traffic  controllers  need  to  be  provided  with  the  necessary  information  to  make 
timely  and  effective  decisions.  In  addition  to  designing  systems  that  provide  the  operator 
with  the  necessary  information,  it  must  be  provided  in  a  cognitively  usable  way  (Endsley 
&  Garland,  2000).  If  infonnation  provided  by  the  tools  overwhelms  controllers’ 
cognitive  capacities,  critical  information  could  be  either  missed  or  misinterpreted  and  put 
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performance  at  risk  (Xing  &Manning,  2005).  For  air  traffic  control,  complexity  reflects 
the  difficulty  involved  in  formulating  an  accurate  representation  of  the  situation,  given 
many  sources  of  information  about  aircraft,  sectors,  and  flight  rules  (Xing  &  Manning, 
2005).  This  is  best  described  as  the  mental  modeling  that  a  controller  must  perform  to 
comprehend  the  aircraft  positions  in  spatial  orientation  to  one  another. 

The  2D  display  does  not  show  altitude  in  an  intuitive  way,  this  interpretation 
becomes  one  of  the  most  mentally  tasking  aspects  of  the  performance  of  controllers’ 
duties.  The  conversion  of  altitude  to  for  a  3D  representation  is  an  important  concept  that 
needs  to  be  considered  when  designing  a  system  in  an  ATC  environment  and  it  must  be 
accomplished  to  develop  full  situation  awareness  (SA).  In  a  study  questioning  student 
controllers  about  this  mental  task  one  stated,  “It’s  like  a  box,  l  guess,  but  it  is  hard  to 
explain,  but  it  is  3D  in  your  head.  In  my  head  it  has  to  be  3D  and  so  ...if  this  is  the 
[aircraft]  itself,  I  imagine  it  being  in  a  3D  box ”,  (Tavanti  and  Cooper,  2009).  Refer  to  the 
Figure  below  that  represents  an  image  of  what  a  controller  attempts  to  create  as  a  mental 
model  of  activity  and  relative  vertical  spatial  orientation  from  the  2D  display. 


Figure  3:  Representation  of  Controller's  Mental  Modeling  of  Flight  paths  (Dang-Nguyen,  2003) 
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A  simple  model  of  an  operator’s  cognitive  task  of  filtering  through  vast  arrays  of 
information  and  detecting  what  is  needed  in  order  to  inform  decision  making  is  illustrated 
in  Figure  4  (Endlsey,  2000).  Ensley  indicates  that  the  amount  of  data  is  immense, 
however  depiction  of  the  data  in  a  form  such  that  it  is  available  to  the  human  needing  it 
can  cause  a  critical  break-down  in  the  information  transfer  which  she  refers  to  is  as  the 
“Information  Gap”.  This  fundamental  concept  is  often  overlooked  in  the  design  of 
human-machine  interfaces.  According  to  Endlsley  (1995),  in  many  systems,  operators 
and  decision  makers  are  bombarded  with  far  too  much  information  to  sort,  leaving  them 
less  informed  as  they  are  unable  to  quickly  access  the  information  they  really  need  from 
the  immense  stream  of  data  in  a  timely  manner.  The  researcher  posits  that  a  more 
intuitive  interface,  that  presents  the  data  intuitively  will  provide  heightened  situation 
awareness  (SA),  permitting  the  operator  to  gain  and  process  the  needed  information  more 
rapidly. 

Data  Produced  Information  Needed 

□  ■ 

ODD 


More  Data  *  More  Information 


Figure  4:  Endsley's  Information  Gap 
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2.5-D,  Slant-View  3D  in  2D,  and  Virtual  Reality 

There  has  been  extensive  investigation  into  the  applicability  of  3D  viewing  for  the 
purpose  of  air  traffic  control.  As  early  as  1948,  papers  were  written  examining  the 
potential  for  3D  displays  of  radar  images  (Parker  and  Wallis,  1948;  Mclntire,  Havig,  and 
Gieselman,  2012).  Numerous  forms  of  radar  image  generation  have  been  investigated  for 
ATC  applications;  including  side-view,  slant  view  and  pilot  perspective.  Below  is  an 


example  display  from  a  recent  experiment  with  slant-view  2D  with  3D  depth  cues. 


Figure  5:  3D  in  2D  Display,  Lateral  Separation  to  Horizon  Obscured  (Bourgois  et  al,  2005) 


An  inherent  limitation  to  this  presentation  is  that  lateral  separation  loses  scale  as 
aircraft  depicted  are  further  away,  (nearer  the  horizon  line),  just  as  in  real  life  the  targets 
become  smaller  and  more  difficult  to  discern  reducing  its  usefulness  for  determining 
separation.  This  limitation  is  recognized  by  Tavanti  et  al,  who  stated,  “visualization 
systems  where  the  users  are  expected  to  discover  relations  based  on  Euclidean  distances 
or  shapes  will  be  ineffective”  (Tavanti,  Le,  and  Dang,  2003). 
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Additionally,  the  side-view  or  slant-range  perspective  induces  far  more  on-screen 
clutter  in  the  form  of  topography,  mapping  and  3D  depth  cues  along  with  overlapping 
data  tags  and  drop  lines  as  aircraft  pass  that  are  more  than  sufficiently  separated  laterally 
increasing  the  difficulty  of  determining  critical  conflicts  as  this  is  3D  in  2D  using  only 
depth  cues  for  reference.,  not  visual  disparity. 

Research  has  demonstrated  that  stereoscopic  display  technologies  are  considered, 
“indispensible”  for  viewing  complex  and  extensive  high-dimensional  scientific  data  and 
objects  especially  in  a  dynamic  and  temporal  (4-D)  capacity  (Chau,  McGinnis  Talandis, 
Leigh,  Peterka  Knoll,  Sumer,  Papka,  and  Jellinek,  2012). ).  However  the  applicability  of 
stereoscopic  viewing  for  the  purpose  of  air  traffic  control  potential  has  been  met  with 
mixed  results  (Parker  &  Wallis,  1948;  Mclntire  et  al,  2012).  In  an  ATC  context  side- 
view  combined  with  a  top- view,  commonly  referred  to  as  “coplanar”,  slant  view  and  pilot 
perspective  stereoscopic  methods  have  been  evaluated.  An  inherent  limitation  to  slant 
view  or  pilot  perspective  stereoscopic  methods  is  that  lateral  separation  loses  scale  as 
aircraft  are  depicted  farther  away  just  as  in  real  life  the  targets  become  smaller  and  more 
difficult  to  discern  at  a  distance.  This  limitation  renders  these  views  useless  for  the 
purpose  of  determining  whether  separation  minima  exist  between  aircraft.  According  to 
Tavanti  et  al,  there  is  a  consistent  deficiency  in  this  type  of  display  with  regards  to  the 
unknown  distance  along  the  depth  axis,  affording  little  to  no  reference  to  the  horizon  line 
(2003). 

Virtual  reality  (VR)  caves  and  other  types  of  semi-immersive  control  interfaces 

have  also  been  investigated  and  were  found  to  have  limitations  when  applied  to  facilitate 

air  traffic  control  functions  (Persiani  and  Liverani,  2000).  Besides  being  physically 
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cumbersome  to  view  by  requiring  complete  body  repositioning  to  view  360  degree 
environments,  they  lack  ground  and  focal  reference  which  can  be  disorienting  to  the  user. 
These  immersive  displays  lack  the  ability  to  provide  a  large  scale  depiction  containing  an 
operator’s  entire  area  of  responsibility  within  a  single  field  of  view  (FOV) 
simultaneously.  Similar  to  the  slant  and  side  views,  they  offer  little  to  no  reference  for 
depth  and  limit  the  viewer’s  line  of  sight  to  one  direction  at  a  time  severely  hindering 
one’s  ability  to  maintain  vigilance  and  obtain  situation  awareness  over  their  entire  area  of 
jurisdiction.  VR  caves  have  potential  for  training  and  orientation,  such  as  learning  the 
limits  of  a  new  facility’s  airspace  or  other  physically  prohibitive  experiences;  however 
their  current  limitations  make  them  an  ineffective  alternative  to  current  day  top-down 
radar  viewing  methods. 

If  stereoscopic  viewing  is  to  be  useful  for  air  traffic  control  operations,  it  must 
simultaneously  depict  lateral  and  vertical  separation.  Recent  advances  in  technology 
allow  for  improved  “top-down”  viewing  of  a  3D  world.  As  a  result,  this  study  focuses  on 
the  effects  of  providing  a  top-down  stereoscopic  3D  view  for  air  traffic  control 
operations.  Such  a  system  maintains  the  perspective  available  to  controller’s  in  today’s 
system  to  allow  the  controller  to  rapidly  grasp  the  lateral  separation,  while  using 
stereoscopic  disparity  to  depict  altitude,  permitting  the  controller  to  assess  the  vertical 
separation  based  on  perception  of  depth  afforded  by  the  stereoscopic  top-down  view. 

The  following  sections  highlight  the  expected  effects  of  incorporating  stereoscopic  3D 
into  traditional  ATC  displays  through  a  top-down  view. 
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Situation  awareness  (SA)  and  Dynamic  Decision-Making 


Situational  awareness  is  the  focus  of  a  controller’s  training  and  the  development 
of  this  skill  as  well  as  the  ability  to  maintain  it  is  critical.  It  is  defined  by  Endsley  as, 
“The  perception  of  the  elements  in  the  environment  within  a  volume  of  time  and  space, 
the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future.” 
(2000).  Refer  to  the  model  by  Endsley  in  figure  6  below  depicting  the  variables 
interacting  through  the  multiple  levels  of  situation  awareness. 


Figure  6:  Endsley's  Model  of  SA  in  Dynamic  Decision  Making  (2000) 

Decision  making,  in  the  context  of  air  traffic  control  systems,  is  a  continuing 
activity  that  utilizes  the  controller's  comprehensive  understanding  of  information  at 
several  levels  of  detail,  together  with  the  controller's  knowledge  of  rules,  procedures  and 
instructions  and  their  permissible  flexibility  (Hopkin,  1995).  The  cognitive  tasks  of  ATC 
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are  what  make  the  profession  a  demanding  one.  The  continual  use  of  memory,  redundant 
call-up  and  execution  of  many  tasks  pertaining  to  a  vast  array  of  differing  situations 
warrant  an  enormous  capacity  for  situation  awareness,  decisive  confidence,  and  keen 
observation  skills. 

The  critical  determinant  of  a  controller’s  performance  is  their  ability  to  detect  the 
need  to  make  corrective  separation  maneuvers,  primarily  by  inferring  information  on 
aircraft  positions  from  a  display.  Signal  Detection  Theory  examines  how  much  more 
tasking  it  is  for  one  to  detect  small  differences  in  large  arrays  or  presentations  of 
seemingly  similar  data  during  visual  search  (1966).  This  is  especially  applicable  to  a 
controller  as  the  variances  that  are  being  tracked,  and  therefore,  manipulated  by  the 
controllers  are  often  very  slight  and  presented  poorly  in  textual  form  with  conventional 
2D  displays.  The  cognitive  task  associated  with  continual  scanning  to  detect  changes  in 
the  form  of  slight  nuances  relates  strongly  to  the  time  to  detect  and  performance  of 
missed  opportunities  for  detection.  Detection  tasks  demand  a  heightened  and  strong 
sense  of  situation  awareness. 

This  researcher  asserts  that  any  tool,  decision  support  system,  (DSS)  or  otherwise, 
that  reduces  a  controller’s  mental  workload  and  potentially  increases  their  SA  is  a  good 
one.  Controllers  are  decision  makers  in  a  dynamic  environment.  Their  success  requires 
constant  updating,  mental  processing  and  application  of  relevant  information  in  the 
interest  of  conflict  detection  and  avoidance  through  the  issuance  of  specific  detailed 
instructions  to  every  pilot  operating  in  their  area  of  jurisdiction. 

It  has  been  found  that  a  controller  has  far  better  recall  about  aircraft  upon  which 

they  provided  more  control  instructions  than  aircraft  that  were  merely  transitioning  their 
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airspace  or  that  they  were  only  monitoring  (Means  et  al,  1988).  This  is  of  particular 
interest  when  one  considers  the  future  of  ATC  and  the  self-separating  systems  that  will 
move  controllers  into  a  passive  role  as  monitors;  this  being  a  similar  capacity  to  the 
manner  in  which  subjects’  skills  were  employed  in  this  study.  There  exists  little  research 
on  how  controller  conflict  detection  will  be  impacted  by  the  passive  monitor  role 
(Metzger,  Domier,  Wessling,  Germany,  and  Parasuraman,  2001).  The  cognitive  effort 
exerted  by  the  controller  in  understanding  certain  characteristics  about  an  aircraft  also 
seems  to  relate  to  their  retention  of  information.  In  a  study  conducted  by  Gronlund,  Bain 
and  Manning  in  1997,  it  was  found  that  controllers  considered  altitude  (83%)  and 
position  (67%)  to  be  the  most  important  factors  to  remember.  They  also  found  that  this 
was  chosen  over  other  vital  flight  information  such  as  route,  destination,  type  aircraft,  call 
sign,  speed  and  time  over  fix.  This  reinforces  the  clear  importance  placed  on  altitude  and 
spatial  orientation  relative  to  one’s  airspace  by  a  controller.  This  result  is  not  surprising 
when  one  considers  their  primary  function  is  ensuring  safe  separation  above  all  else. 

The  following,  Figure  7,  is  an  example  using  simple  3D  (2.5D)  graphics  of  the 
spatial  orientation  of  two  aircraft  in  a  cylindrical  section  of  airspace  with  boundaries 
depicted  as  transparent.  This  is  the  type  of  mental  modeling  from  a  2D  display  top-down 
presentation  using  textual  altitude  codes.  The  mental  image  creation  is  tasking  for  a 
controller  and  this  example  illustrates  how  it  is  employed,  although  cues  such  as  drop 
lines,  space  boundary  and  vertical  scale  are  not  imagined — they  are  comprehended.  As 
this  scene  changes,  based  upon  instructions  given  and  executed,  the  controller  must 
constantly  update  their  3D  mental  model  to  permit  a  clear  understanding  of  all  the  aircraft 

under  his/her  authority  and  maintain  that  separation. 
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Toggle  View  1 


Toggle  View  2 


Figure  7:  Coplanar  3D  in  2D  view  of  2  aircraft  with  depth  cues  (Thomas  and  Wickens,  2006) 


Figure  8  shows  a  simplified  side  view  of  several  aircraft  flight  paths  at  varied 
altitudes  and  varied  changes  in  altitude  and  their  impacts  on  conflicts  based  on  timing. 
This  concept  has  been  experimented  with  EUROCONTROL  using  what  is  called 
coplanar  workstations  that  offer  a  top-down  traditional  view  as  well  as  a  type  of  “side- 
view”  to  determine  vertical  spacing.  However,  the  drawback  of  dividing  one’s  attention 
between  two  large  displays  as  well  as  attending  to  duplicates  of  aircraft  tracks  is  the 
potential  for  error  as  operators  must  attend  to  their  primary  presentation  at  all  times, 
making  the  introduction  of  peripheral  displays  a  potential  distraction. 
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Figure  8:  Slant-range  and  Side-View  Examples  from  SIGNAL  software 


Creating  a  stereoscopic  3D  view  of  traditional  top-down  digital  radar  displays 
affords  the  controller  the  best  possible  perspective,  preserving  both  the  scale  and  lateral 
separation  while  providing  a  scale-accurate  depth  perspective  to  assess  vertical 
separation.  By  providing  simultaneous  vertical  and  lateral  separation,  it  is  hypothesized 
that  situational  awareness  will  be  enhanced  resulting  in  the  following  first  hypothesis: 

HI:  The  use  of  stereoscopic  3D  digital  radar  displays  will  have  a  positive  effect 
on  controller  situation  awareness  and  conflict  detection. 
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Mental  Modeling,  Memory  and  Recall 

The  role  of  recall  and  memory  in  ATC  is  imperative  for  certain  duties.  Recalling 
rules  and  regulations,  potential  outcomes  of  a  given  scenario  from  experience,  or 
knowing  how  an  input  will  interact  with  a  dynamic  traffic  flow  are  all  critical  abilities  for 
air  traffic  controllers.  However,  actual  memory  of  a  specific  situation  has  positives  as 
well  as  drawbacks. 

Hopkin  suggested  that  a  controller’s  ability  to  forget  the  last  altitude  or  vector 
given  to  an  aircraft  can  permit  them  to  replace  that  data  with  the  current  or  next  assigned 
characteristics  (1980).  This  memory  management  may  prevent  older  information  from 
interfering  with  the  newest  information,  permitting  a  controller  to  maintain  better 
awareness  of  their  traffic  (Gronlund  et  al,  1997).  Further  research  has  evidenced  that  the 
mental  refresh  rate  can  contribute  more  to  the  misperception  of  the  true  model  as  what  an 
operator  understands  to  be  happening  becomes  out-dated  information.  Fundamentally, 
the  ability  to  present  information  and  then  permit  its  easy  replacement  with  updated  data 
may  be  more  desirable  in  ATC  (NATO,  1982:36). 

In  a  2004  paper  presented  to  the  International  Journal  of  Aviation  Psychology,  a 

controller’s  cognitive  task  was  defined  as; 

Given  the  dynamic  nature  of  ATC,  the  temporal  aspects  of  controllers’  mental 
picture  are  apparent,  as  are  the  temporal  demands  of  their  tasks:  Controllers 
need  to  anticipate  aircraft  trajectories  and  pilot  intentions  well  into  the  future, 
plan  their  actions,  and  then  execute  the  planned  actions  at  a  proper  time  and  in 
an  appropriate  sequence.  (Rantanen,  McCarley  and  Xu,  2004) 

This  is  commonly  referred  to  as  “The  Picture”  by  air  traffic  controllers.  In  fact, 

when  one  controller  relieves  another,  it  is  the  responsibility  of  the  one  being  relieved  to 

ensure  the  on-coming  controller  has  a  satisfactory  concept,  or  “Picture”  of  the  traffic 
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before  departing  the  workstation  (FAA,  2012).  Steven  Shorrock  has  written  much  on  air 
traffic  controllers’  mental  picture  and  its  importance,  (eg.  Mental  Imagery  in  Air  Traffic 
Control,  2010;  Errors  of  Perception  in  Air  Traffic  Control,  2007).  He  states  that  there  is 
an  absence  of  emphasis  on  this  important  ATC  concept  and  the  implications  of  learning 
imagery,  perceiving  it  and  adapting  it  into  technology  design  are  significant.  He  also 
indicates  that  mental  imagery  is  a,  “controversial  subject  in  psychology”  albeit  an 
imperative  one  to  understand  in  the  matters  of  ATC  performance  measures  (2010). 

As  with  any  mental  formulation  or  processing  of  information,  whether  it  is 
memory  of  pictures,  scenes,  activities,  faces  or  text,  the  accuracy  lies  in  one’s  ability  to 
perceive  the  information  as  intended.  The  ability  of  the  controller  to  take  2D 
presentations  of  aircraft  in  flight  and  mentally  formulate  a  complex,  dynamic  mental 
model  is,  “a  vital  cognitive  component  of  their  job”  (Shorrock  and  Isaac,  2010).  This 
leads  one  to  believe  that  any  mistakes  in  perception  can  lead  to  mistakes  when  decisions 
about  aircraft  separation  are  made  based  upon  these  mental  models.  Jones  and  Endsley 
found  that  76%  of  pilot  errors  were  a  direct  result  of  misinterpreting  presented  and 
available  information  (1996). 

This  makes  the  heightened  awareness  of  activity  pivotal  to  ensuring  the  mental 
image  accuracy  in  dynamic  environments  such  as  ATC  (Endsley,  2000).  Endsley  goes 
on  to  infer  that  mental  models  are  imperative  tools  for  SA  building  and  maintenance.  She 
indicates  that  they  are  a  method  for  unifying  data  into  usable  projections  to  anticipate 
future  events  (Endsley,  2012)  and  that  “Errant  Mental  Models”  can  result  from  missing 
an  integral,  although  minute,  detail  that  has  changed  therefore  altering  one’s  entire 

concept.  This  could  be  a  course  variation,  or  slight  delay  in  a  climb  or  descent. 
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Given  the  importance  of  the  mental  model  to  air  traffic  controller  performance 
and  the  effect  of  recall  accuracy  on  establishing  a  reliable  mental  model,  hypothesis  2  is 
as  follows: 

H2:  The  use  of  stereoscopic  3D  digital  radar  displays  will  have  a  positive  effect 
on  controller  recall  accuracy  of  aircraft  vertical  position. 

Perceptions  of  Fatigue 

Of  human  factors  studies  in  the  aviation  industry,  few  are  more  thoroughly 
examined  or  regarded  as  important  as  that  of  fatigue  management.  There  is  a 
preponderance  of  literature  examining  aircrew  and  controller  fatigue  issues  to  include 
circadian  rhythm,  work  and  rest  cycles,  and  crew  rest  period  interruptions  among  others. 
Fatigue  can  be  defined  as,  “. .  .a  decrease  in  performance  capability  as  a  function  of  time 
on  task”  (Salas  and  Maurino,  2010).  Unfortunately,  in  this  researcher’s  observation,  this 
is  due  largely  to  the  fact  that  fatigue  is  the  most  frequently  cited  reason  for  major  aviation 
mishaps. 

There  are  many  aspects  of  ATC  training  and  procedures  designed  for  the  purpose 
of  fatigue  management  and  almost  all  involve  dedicated  rest  periods,  crew  resource  and 
workload  management  of  some  sort.  However,  the  predominant  existing  research 
concerning  fatigue  focuses  upon  the  physical  wakefulness  and  time  at  and  away  from  the 
duty  location  and  workstation;  not  the  impact  of  the  technologies  employed  to  assist  the 
controller  in  the  performance  of  their  duties. 

In  this  study,  the  fatigue  focus  is  on  that  which  is  induced  through  cognitive  tasks 

and  workload  associated  with  the  use  of  the  presentations  from  which  controllers  derive 
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their  mental  models.  Controllers  have  an  unusually  high  intensity  of  demands  on  their 
senses  having  to  process  vast  amounts  of  data  often  for  durations  of  two  hours  or  more  at 
a  time  (Shorrock,  2006). 

Certainly  the  fatigue  issues  of  controllers  are  not  the  same  as  long-haul  pilots  who 
fly  12  hour  trans-meridian  flights;  however,  there  are  basic  task-specific  considerations. 
Of  primary  concern  is  that  of  the  controller’s  ability  to  maintain  vigilance  when  the 
workload  is  low.  It  has  been  published  that  there  is  a  consistent  reduction  in  controllers’ 
signal  detection  efficiency  following  a  significant  workload,  (surge  of  traffic),  resulting 
in  most  operational  mistakes  taking  place  under  light  to  moderate  workloads  (Redding, 
1992).  Perhaps  improved  human-computer  interfaces  and  intuitive  presentation  of 
information  can  reduce  this  trend. 

Given  the  cognitive  nature  of  the  tasks,  it  stands  to  presume  a  more  intuitive 
display  that  allows  a  controller  a  more  realistic  depiction  of  the  activities  in  their  airspace 
will  minimize  their  mental  task-load  and  therefore  slow  the  onset  of  fatigue  resulting  in 
the  following  hypothesis: 

H3:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller 

fatigue. 

Perceptions  of  Workload  and  Task  Difficulty 

It  has  been  extensively  studied  and  agreed  upon  by  many  in  the  human 

performance  field,  that  temporal  and  mental  demands  comprise  the  largest  portion  of  a 

controller’s  workload  in  ATC  (Rantanen,  McCarley,  and  Xu,  2004).  The  limiting  factor 

to  the  growth  of  the  air  traffic  volume  is  also  known  to  be  the  air  traffic  controllers’ 
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limitations  on  monitoring  and  controlling  separation  of  traffic  within  each,  increasingly 
saturated,  airspace  sector.  It  has  been  determined  as  recently  as  2004  that  the  maximum 
safe  operating  number  of  active  aircraft  per  operator  is  15  at  a  time  (Erzberger,  2004).  In 
the  author’s  experience,  this  is  a  significantly  high  workload  by  any  controller’s 
standards  and  perhaps  unrealistically  high  to  maintain  for  any  substantial  duration. 

As  cited  earlier  in  this  report,  the  primary  contributor  to  workload  in  the 
controller’s  task  is  the  3D  mental  model  creation  from  the  2D  presented  information.  In 
addition  to  this  workload  is  the  labor  intensive  manner  in  which  current  operations  are 
conducted,  namely  in  controller-to-pilot  communications  for  verification  purposes.  An 
example  of  this  is  the  needed  number  of  radio  transmissions  to  ensure  the  pilot  and 
controller  understand  each  other  and  instructions  are  received  and  adhered  to.  In  a  study 
of  communication  delays,  it  was  explained  that  when  a  controller  is  attempting  to  verify 
an  altitude  due  to  a  data  tag’s  temporary  absence  or  a  rapidly  changing  situation,  he  or 
she  will  have  to  query  a  pilot  and  await  the  response.  The  pilot  will  respond,  generating 
another  question  or  instruction  issued  by  the  controller  and  then,  again,  waits  for  a  pilot 
response  (Rantanen,  McCarley,  and  Xu,  2004).  This  is  an  inefficient  method  and  could 
seemingly  reduce  the  questions  a  controller  may  have  about  aircraft  activities  as  it  will  be 
displayed  very  apparently  in  the  3D  model  provided  by  the  display.  This  leads  the 
investigator  to  the  following  additional  two  hypotheses: 

H4:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller 
perceived  workload. 

H5:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller 


perception  of  task  difficulty. 
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Perceived  Display  Induced  Distraction 

Tavanti  and  Cooper  discovered  that  the  3D  technology  itself  presents  the  needed 
information  in  such  an  intuitive  manner  that  it  poses  little  distraction  for  the  users  (2009). 
This  finding  coincides  with  additional  research  done  in  the  field  of  auto  stereo  3D 
desktop  applications  for  scientists  where  the  ability  to  focus  on  the  science  problem 
presented  without  distraction  has  been  the  focus,  presenting  that  field  with  seamless  and 
non-intrusive  new  display  technology  (Chau  et  al,  2012).  In  Chau’s  study,  the 
investigators  determined  that  stereoscopic  presentations  of  data  permitted  the  user  to 
focus  their  attention  quickly  on  the  specific  information  they  sought  more  readily  even  on 
large  congested  and  “cluttered”  displays. 

This  relates  very  well  to  ATC  where  it  has  been  found  that  much  of  a  controller’s 
time  is  spent  making  superfluous  “scans”  of  the  screen  and  observing  and  “re-visiting” 
aircraft  tracks  that  are  of  no  immediate  concern.  Perhaps  by  presenting  the  tracks  with 
depth,  their  vertical  separation  will  be  obvious  enough  that  a  controller  will  be  able  to 
“filter”  through  the  traffic  that  is  operating  in  safe  proximity  and  quickly  zero-in  on  those 
with  less  time  to  loss  of  separation  (LOS)  enhancing  detection  while  minimizing 
distraction. 

Another  important  performance  aspect  is  that  of  short-term  and  “prospective” 
memory  (Wickens,  2002).  This  type  of  memory  is  specific  to  remembering  that 
something  needs  to  be  done  in  the  near  future  and  is  considered  a  very  poor  human 
characteristic  in  multi-tasking  contexts  (Wickens,  2002).  This  can  be  attributed  to  the 


high  potential  for  distractions  that  may  divert  attention.  This  can  be  best  illustrated  by  a 
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controller  assigning  an  altitude  to  a  departing  aircraft  much  lower  than  the  aircraft’s 
requested  level  due  to  other  traffic  precluding  the  immediate  climb.  A  controller  will 
often  utilize  a  tactic  such  as  “cocking  the  strip”  where  they  will  literally  position  a  paper 
flight  strip  associated  with  that  flight  in  a  particular  manner  as  to  assist  them  in  recalling 
what  to  do  when  they’re  able — sort  of  a  string-on-the-finger  memory  jogger. 
Unfortunately,  these  tactics  are  workload  permitting  and  are  inconsistently  utilized 
resulting  in  inefficiencies  such  as  a  pilot  being  held  low  for  longer  than  required  because 
a  controller  forgot  to  eliminate  the  restriction  and  permit  the  climb.  If  one  were  to 
imagine  this  sort  of  mistake  on  a  magnitude  of  thousands  of  times  a  day,  even  when  a 
climb  to  a  more  efficient  fuel-bum  altitude  is  delayed  even  for  a  minute  or  two,  perhaps 
millions  of  dollars  in  fuel  expense  could  be  realized.  This  leads  to  the  final  hypothesis: 

H6:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller 

distraction. 

Supporting  Theory 

Channel  Expansion  Theory 

A  highly  regarded  and  very  useful  theory  in  the  discipline  of  human-computer 
interaction  and  information  technology  is  that  of  Channel  Expansion  by  Carlson  in  1995. 
This  particular  theory  concentrates  on  the  importance  of  a  channel  or  media’s  richness,  or 
its  ability  to  convey  meaningful  and  clear  information  as  an  information  technology. 

This  is  certainly  applicable  when  investigating  the  impacts  of  new  IT  for  use  in  a  time- 
critical  decision-making  environment  such  as  ATC.  Carlson  and  Zmud  indicate  how 
individuals  develop  perceptions  of  a  medium’s  ability  to  provide  rich  communications  as 
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dynamic,  changing  and  developing  over  time  with  exposure  to  the  computer-mediated- 
communication,  (CMC).  Their  findings  reveal  that  people’s  perceptions  vary  over  time 
with  their  usage  and  with  exposure  and  users  may  hold  varied  levels  of  confidence  in  a 
channel’s  ability  to  function  specific  to  the  applications  they  desire  (Carlson  and  Zmud, 
1999).  This  theory  is  simple  in  the  fact  that  it  refers  to  experiential  impacts  on 
perception,  however,  humans  naturally  have  stereoscopic  sight  with  overlapping  visual 
fields  that  afford  the  depth  disparity  people  are  intuitively  able  to  decipher.  This  makes 
the  inherent  “richness”  of  information  displayed  stereoscopically,  as  a  CMC  information 
technology,  one  of  immediate  interest  as  individuals  who  see  it  are  able  to  quickly 
understand  what  is  being  shown  as  it  provides  a  realistic  feeling  of  depth  and  space. 

Summary 

There  is  a  vast  amount  of  literature  concerning  the  implications  of  3D  displays  in 
ATC — its  advantages  and  drawbacks  in  many  “unrealistic”  and  varied  manners. 
However,  there  is  little  empirical  evidence  found  in  the  literature  on  the  specific  human 
performance  factors  and  common  measurements  of  a  controller’s  performance  with 
regard  to  the  passive  monitoring  role  in  a  current-day,  realistic,  real-time,  dynamic  traffic 
environment  or  even  within  the  confines  of  current  standards  of  operation.  This  study 
will  attempt  to  begin  filling  this  gap  in  the  prevailing  literature  on  3D  use  for  ATC  with 
empirical  evidence  gathered  from  the  observation  and  responses  of  35  USAF  controllers. 

Table  1  below  outlines  the  hypotheses  generated  in  this  section  derived  from  the 
literature  review  and  the  researcher’s  experience  in  the  field. 
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Table  1:  Hypotheses 


HI:  The  use  of  stereoscopic  3D  digital  radar  displays  will  have  a  positive  effect  on 
controller  situation  awareness  and  conflict  detection. 

H2:  The  use  of  stereoscopic  3D  digital  radar  displays  will  have  a  positive  effect 
on  controller  recall  accuracy  of  aircraft  vertical  position. 

H3:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller  fatigue. 

H4:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller  perceived 
workload 

H5:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller  perception 
of  task  difficulty. 

H6:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease  controller  distraction. 
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III.  Methodology 


Overview 

The  preceding  chapters  identified  the  potential  of  stereoscopic  display  technology 
benefits  in  the  complex,  multi-tasking,  high-speed  decision  critical  environment  of  ATC. 
This  chapter  will  present  the  methodology  used  to  investigate  the  research  hypotheses 
proposed  in  Chapter  II.  Included  are  descriptions  of  the  research  design,  pilot  study, 
population  of  study,  survey  instrument,  data  collection,  and  the  actual  experiment  setup 
and  execution. 

Relevant  Population 

This  research  remains  applicable  to  the  entire  air  traffic  control  population. 
However,  for  the  purposes  of  this  study,  the  sample  population  was  provided  by  the  ATC 
technical  training  school  at  Keesler  Air  Force  Base,  MS,  as  it  had  been  previously 
determined  the  best  location  to  conduct  the  experiment  with  the  leadership's  support.  It 
was  carried  out  using  35  USAF  ATC  instructors  and  students  including  both  civilian  and 
active  duty.  Participation  was  voluntary  and  the  study  lasted  one  week.  All  the 
participants  were  familiar  with  the  air  traffic  procedures  used  in  the  experiment  as  they 
were  derived  from  current  standard  practices  and  were  in  line  with  FAA  and  AFI 
guidance.  To  avoid  compromising  the  study,  participants  were  asked  not  to  share  the 
details  of  the  study  or  talk  about  the  questionnaires  with  others  until  the  end  of  the  data 
collection  period. 
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Experiment  Design  Considerations 

The  experiment  was  designed  to  capture  perceptions  of  the  technology  by  the 
users  as  well  as  test  for  efficiency  improvements  during  the  experiment.  All  the 
participants  were  administered  a  brief  training  scenario,  a  control  condition  of  2D  and  an 
experimental  stereoscopic  3D  condition. 

All  scenarios  had  to  be  recorded  prior  to  their  exposure  to  the  participants  thereby 
limiting  the  subjects’  interaction — this  is  identified  as  a  limitation  of  the  design.  Based 
on  the  timeline  and  budgetary  constraints,  this  was  a  necessary  adaptation.  This  was  also 
found  to  be  quite  acceptable  as  the  scenarios  remained  the  same  for  each  subject  since 
controllers  were  unable  to  modify  them  with  their  own  inputs.  A  real-time  interactive 
scenario,  such  as  the  standard  trainer  simulations  controller’s  use  that  require  the 
controller  to  issue  instructions  which  are  then  input  by  a  pseudo-pilot,  would  have  been 
far  more  complex  to  quantify  and  code  results  as  each  operator  would  perform  the  tasks 
of  separation,  (through  heading  and  altitude  issuance),  very  differently,  employing 
personally  preferred  techniques.  However,  by  maintaining  the  exact  same  scenario 
across  all  participants  by  way  of  prohibiting  instructions  that  would  alter  the  traffic  flow, 
the  specifically  scripted  conflicts  were  presented  in  the  exact  same  manner  to  all 
participants  allowing  consistent  data  collection  through  the  dynamic  feedback  measure  of 
conflict  detection. 

This  experiment  was  designed  to  evaluate  the  controller  recognition  of  vertical 
separation  conflicts  in  a  3D  environment  over  that  of  a  standard  2D  presentation.  The 
experiment  was  run  first  through  a  brief  pilot  study  with  six  volunteer  participants  to 
ensure  validity.  It  was  also  evaluated  by  experienced  air  traffic  control  volunteers  to 


33 


evaluate  realism,  quality  and  concurrency  with  the  apprentice  subjects’  level  of 
proficiency  and  ability. 


Equipment 

For  stereoscopic  display  and  simulated  scenario  creation  and  modification, 
numerous  pieces  of  varied  equipment  was  procured  and  manipulated  to  achieve  the 
desired  presentation  for  testing.  Of  note,  all  components  of  this  experiment  were 
conducted  with  commercial-off-the-shelf  technology  with  the  exception  of  the  SIGNAL 
FAA  simulation  software,  which  is  copyright  and  government  protected.  Included  in  this 
final  suite  was  the  array  of  software  and  hardware  listed  here: 

1.  Lenovo  Thinkpad  T420i  ,  64  bit,  intel  core  i3-2310M  processor,  Windows  7 
Enterprise  OS,  4.00GB  memory,  300  GB  HD.  Used  for  scenario  generation  using 
SIGNAL  FAA  provided  simulation  software.  This  device  acted  as  a  slave  to  the 
NVIDIA  equipped  Dell  that  captured  video  as  it  was  generated 

2.  SIGNAL  Air  Traffic  Control  Simulation  Program  software.  Authorized  use  by 
FAA  to  develop  and  record  air  traffic  scenarios 

3.  HP  computer  with  Clickcounter  freeware  that  was  used  remotely  for  mouse  click 
count  (conflict  detection) 

4.  Microsoft  wireless  mobile  mouse  3500  (2).  Used  by  the  participants  for  subjects’ 
real  time  inputs  and  used  by  the  researcher  to  load  the  scenes  on  the  monitor 

5.  Epiphan  DVI2USB  Duo  External  DVI  Capture  box  device.  Used  to  record  the 
simulated  ATC  scenarios  from  the  originating  machine  (Lenovo  laptop)  to  the 
receiver  and  recorder  machine  (Dell  laptop) 

6.  Dell  precision  M4600  workstation,  Windows  Professional  7,  64  bit,  250  GiG  HD, 
16  GB  DDR3  SDRAM,  NVIDIA  3D  suite,  NVIDIA  Quadro  1000M  discrete 
graphics  card,  intel  core  i7  extreme  processor.  It  was  used  to  capture  the  ATC 
scenes  and  later  connected  to  the  main  display  that  was  viewed  by  the  participants 

7.  StarTech  display  port  to  DVI  Dual  link  Active  Converter  cable-USB  powered.  It 
was  used  to  connect  the  Dell  computer  to  the  ASUS  monitor  to  facilitate 
operating  NVIDIA  3D  active  shutter  display. 

8.  ASUS  VG236  23.5"  3D  120Hz  LCD  display  monitor  (commercial). 
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9.  NVIDIA  3D  software  used  to  load  the  ATC  scenes  that  were  showed  for  the 
participants 

10.  3D  vision  USB  Controller/IR  Emitter.  It  is  a  needed  component  to  NVIDIA 
software 

All  the  scenarios  were  developed  using  the  Signal  software.  They  were  recorded 
from  the  Lenovo  computer  using  the  capture  box  (commonly  referred  to  as  a  “frame 
grabber”)  to  the  Dell  computer,  which  was  connected  to  the  ASUS  3D  Monitor.  The 
scenarios  were  shown  to  the  participants  on  the  23.5”  ASUS  3D  monitor  using  the 
NVIDIA  3D  viewer  in  stereoscopic  with  active  shutter  LCD  glasses  to  provide  each  eye 
its  specific  view  of  the  images. 

Lor  the  experiment,  three  scenarios  were  recorded:  training,  control,  and 
experimental.  The  training  scenario  was  developed  to  allow  the  controllers  to  get 
familiarized  with  the  experimental  process  and  with  what  the  study  focused  on  prior  to 
collecting  measurements.  However,  they  were  not  presented  with  the  new  technology 
until  the  experimental  scenario.  The  control  scenario  was  the  baseline  to  compare 
controllers’  performance,  although  it  was  not  necessarily  their  first  exposure.  The 
experimental  scenario  was  displayed  in  full  stereoscopic  3D  with  the  peripheral  headgear 
of  wireless  IR  active  shutter  3D  glasses.  This  was  the  participants’  first  exposure  to  the 
3D  presentation  and  a  20-second  delay  was  built  into  each  scenario  to  allow  their  eyes  to 
adapt  to  the  stereoscopic  view  before  measurable  activities  commenced.  Verification  of 
this  adaptation  was  done  by  verbal  questioning  to  verify  they  indeed  were  seeing  a 
disparity  view. 
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Questionnaire  Development 

Please  refer  to  Appendices  F,  G,  and  H  for  actual  copies  of  the  questionnaires 
utilized  in  this  study  to  include  demographic  pre-screening  and  post-  control  and 
experimental  treatment  instruments.  These  were  varied  in  their  formats  to  permit  the 
collection  of  a  diverse  amount  of  data  from  responses.  The  Pre-Screening  Questionnaire 
and  training  questionnaires  offered  fill-in  the  blank,  YES  and  NO,  and  Likert  scales  of  1 
-  5  for  a  number  of  responses  with  1  equating  to  “strongly  disagree”  and  5  “strongly 
agree”. 

The  Post-Treatment,  (control  and  experimental)  Questionnaires  contained  a 
screen  capture  of  the  radar  scope  from  a  time  two  minutes  prior  to  the  end  of  the  scenario 
that  prompted  the  participants  to  fill-in  the  altitudes  from  memory,  using  spatial  reference 
as  an  aid  to  recall,  see  Figure  9  below  for  an  example  of  these  captures. 
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Figure  9:  Screen  Caption  from  Post-Treatment  Questionnaire 


The  limited  data  bocks  with  the  call  signs  remained  in  the  screen  capture  as  it  is 
known  controllers  do  not  remember  call  signs  well  (Gronlund  et  al.,  1997).  It  also  had  a 
number  of  fill-in  the  blank  questions  regarding  extremes  observed  in  the  scene  such  as 
highest  aircraft,  what  conflicts  were  evident  and  approximations  of  vertical  separation. 
Likert  scales  of  1  -  7  were  also  used  with  1  equating  to  a  “No”  and  7  a  “Yes”  with  4 
coded  as  “Somewhat”.  Additionally,  there  were  two  questions  asked  about  the  feasibility 
of  the  technology  and  their  perceptions  of  it.  This  provided  candid  and  frank  feedback 
from  the  individuals  on  their  observations  regarded  as  valuable  insight  by  this 
investigator. 

Currently,  there  is  no  general  agreement  upon  a  correct  methodology  to  measure 
SA  (Gronlund  et  al,  1997),  however  the  query  technique  is  the  most  commonly  used 
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method  as  it  permits  one  to  recall  what  they  are  able  to  from  the  point  in  time  they 
actually  possessed  situation  awareness  (Adams,  Tenney,  and  Pew,  1995).  This  is  a 
flawed  manner  in  which  to  measure  SA,  however  as  an  accepted  limitation,  remains  the 
most  viable  method  for  this  type  of  measurement  that  precludes  the  actual  determination 
of  one’s  actual  level  of  awareness  whilst  they  perform  the  task  functions  in  real-time. 

Pilot  Study 

A  pilot  study  was  carried  out  approximately  three  weeks  before  the  initial 
experiment.  The  pilot  study  was  conducted  with  six  volunteers,  from  which  two  were  Air 
Force  Institute  of  Technology  graduate  students,  three  were  air  traffic  controllers  from 
Wright  Patterson  AFB  Tower  and  one  was  a  human  factors  researcher  and  ex-airline 
pilot.  To  test  both  the  experiment  design  for  flow  and  timing  as  well  as  mobility,  it  was 
conducted  in  the  researcher’s  lab  on  The  Air  Force  Institute  of  Technology  campus  and 
was  packed  and  transported  to  the  Air  Force  Research  Laboratory  also  located  on  Wright  - 
Patterson  AFB,  Dayton,  OH. 

Goals  of  the  pilot  study  were  to  ensure  that  the  format  and  techniques  established 
in  the  experiment  design  would  permit  the  collection  of  the  information  sought.  It  was 
also  used  to  ensure  that  the  timeline  and  script  order  were  accurate  for  actual  real-time 
administration  of  the  study.  Additionally,  It  was  also  used  to  determine  that  an  intended 
participant  would  be  capable  of  understanding  what  was  being  presented  well  enough  to 
provide  useful  post-treatment  feedback  and  that  the  difficulty  posed  a  challenge,  yet 
remained  realistic.  This  created  confidence  in  the  design’s  ability  to  ascertain 
performance  measures. 
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Useful  feedback  and  observations  were  received  from  the  pilot  study  participants. 
Some  identified  the  intensity  of  the  disparity  to  cause  near-immediate  eye  strain;  this  was 
later  determined  to  be  caused  by  the  z-scale  adaptation  being  too  drastic.  Adjustment  of 
this  parameter  from  30%  disparity  on  the  z-axis  scale  to  20%,  resulted  in  a  marked 
improvement  in  reported  viewer  eye  comfort. 

Also,  due  to  the  lack  of  training  on  the  new  technology  and  little  preparation  for 
what  the  technology  would  look  like  or  what  sort  of  questions  they  may  be  asked  at  the 
conclusion,  study  participants  were  completely  unprepared  to  answer  post-treatment 
questionnaires  as  they  were  written.  The  format  of  a  fill-in-the-blanks  chart  indicating  a 
side  view  from  the  surface  up  to  the  highest  displayed  aircraft  proved  to  be  too  difficult 
for  the  participants  to  recall.  This  instrument  method  was  abandoned  in  favor  of  a 
modified  screen  capture  allowing  the  controllers  to  rely  on  the  relative  horizontal  position 
of  a  target,  (as  described  briefly  in  the  previous  section,  “Questionnaire  Development”), 
to  recall  its  approximate  vertical  orientation  on  the  scope. 

Due  to  the  difficulties  participants  encountered  with  the  pilot  study’s  format,  no 
usable  data  was  collected  which  was  warranted  acceptable  since  the  purpose  was  to 
indeed  detect  these  deficiencies  and  correct  them.  Several  changes  were  made  between 
subjects  of  the  pilot  study  so  that  the  last  volunteer,  the  sixth  to  run  it,  was  exposed  to  the 
actual  experiment  as  it  was  fielded.  Among  other  modifications  to  the  experiment 
through  the  pilot  study,  a  very  brief  training  segment  was  devised  and  inserted  as  the  first 
exposure  for  every  participant  in  each  segment  of  the  experiment  to  better  prepare  them 
for  the  experiment’s  procedures  and  purpose. 
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Cooperation  and  Permission  by  the  USAF  ATC  Field 

This  experiment  involved  the  performance  of  Air  Force  personnel,  both  civilian 
and  active  duty,  and  was  conducted  in  accordance  with  human  experimentation 
requirements  (AFI  40-402).  Exemption  to  full  Institution  Review  Board  was  sought  in 
order  to  use  Department  of  Defense  personnel  as  volunteers  for  research  on  3D 
stereoscopic  radar  displays  and  the  potential  improvement  in  controller  performance  with 
them.  This  exemption  was  granted  by  the  Air  Force  Institute  of  Technology  on  25 
January,  2012  by  the  AFIT  IRB.  Additional  authorization  to  access  active  duty  US  Air 
Force  and  DoD  Civilian  controllers  at  the  technical  school  was  granted  by  USAF  ATC 
Career  Field  Manager,  AF  Pentagon  Washington  D.C.  and  the  school’s  commander  at 
Keesler  AFB,  MS  on  7  January,  2012.  Please  refer  to  Appendices  A  and  B  for  these 
authorization  letters. 

The  Experiment 

For  this  study  the  sample  population  was  provided  by  the  ATC  technical  training 
school  at  Keesler  Air  Force  Base,  MS.  The  study  involved  35  USAF  ATC  instructors  and 
students.  Participation  was  voluntary  and  the  study  lasted  one  week.  A  primary 
advantage  of  this  population  was  that  all  participants  were  familiar  with  standard  air 
traffic  procedures  and  displays.  Experience  levels  varied  from  2  months,  to  30  years. 

The  average  years  of  ATC  experience  was  7.29  years.  The  average  age  of  participants 
was  29.1  years  and  26  of  the  35  participants  were  male. 

The  subjects  were  volunteers  and  were  available  for  up  to  two  hours  of  time. 
Participants  were  briefed  before  starting  the  experiment  about  each  section  of  the 


40 


simulation  and  a  pre-questionnaire  was  administered  to  gather  pertinent  participant 
demographics  They  were  asked  to  monitor  two,  multiple- aircraft  radar  scenarios  in  both 
2D  traditional  and  a  3D  stereoscopic  display  configuration.  All  the  scenarios  were 
viewed  in  a  darkened  room  to  best  reduce  the  illumination  loss  from  the  active  shutter 
glasses  as  well  as  simulate  a  realistic  radar  approach  control  room  setting.  The  scenarios 
were  designed  to  be  as  realistic  as  possible  in  order  to  test  the  hypotheses  of  interest.  The 
scenarios  included  a  mean  of  eight  aircraft,  two  departures;  two  aircraft  had  no 
supplementary  data  tags— an  entirely  realistic  scenario  in  actual  ATC  operations. 

The  SIGNAL  software  provided  familiar  training  simulations  using  a  simulated 
terminal  environment  created  specifically  for  use  by  the  USAF  ATC  technical  school 
where  the  experiment  was  conducted.  This  simulated  airport  is  called,  “Canyon”  and  was 
very  familiar  to  all  participants  and  chosen  specifically  for  this  reason.  The  mapping  and 
type  of  aircraft  and  the  characteristics  of  these,  as  well  as  their  typical  call  signs,  were 
used  for  familiarity  by  the  subjects.  This  pseudo  airport  was  used  to  generate  the  entire 
set  of  scenario  recordings  used  which  was  recorded  ahead  of  time  to  permit  both 
stereoscopic  conversion  and  maintain  control  over  the  presentations  for  uniformity. 

The  experimental  3D  scenarios  were  presented  on  the  ASUS  23.5”  monitor,  (see, 
“Equipment”)  with  the  targets  shown  as  “protruding”  to  draw  attention  to  and 
exaggerating  their  vertical  separation  in  addition  to  their  lateral  spacing.  This  was  done 
using  stereoscopic  viewing  through  left  eye/right  eye  disparity  and  channeled  using  the 
NVIDIA  3D  vision  active  shutter  viewing  glasses  and  IR  emitter.  The  control  scenarios 
were  also  presented  on  the  same  display  without  the  peripheral  NVIDIA  3D  equipment 
activated. 
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The  workstation  developed  and  used  for  used  for  the  experiment  can  be  seen  in 
Figure  10.  The  researcher  expected  the  subjects’  performance  to  increase  for  detecting 
true  vertical  separation  conflicts  and  altitude  deviations  when  observing  traffic  in  a  3D 
presentation.  It  was  anticipated  that  the  reduction  in  supplementary  information  or  the 
need  to  read  and  calculate  constant  altitude  readouts  to  produce  a  3D  mental  image  will 
reduce  mental  fatigue  thus  allowing  controllers  to  maintain  SA  for  longer  durations 
before  tiring,  this  being  suggested  and  supported  by  the  literature.  It  was  also  anticipated 
that  a  certain  number  of  subjects  would  respond  poorly  to  the  3D  images  as  this  type  of 
viewing  can  induce  headaches,  dizziness  or  disorientation  in  certain  individuals  who  have 
poor  stereopsis.  To  mitigate  this  danger,  a  brief  questionnaire  and  two-minute  exposure 
was  conducted  prior  to  the  experiment  for  each  subject. 


Figure  10:  Experimental  Workstation  in  Use 
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Participants  were  told  in  advance  that  the  scenarios  would  be  stopped  after  each 
scenario  that  they  would  be  asked  detailed  questions  about  what  they  observed. 
Controllers  were  asked  to  actively  monitor  the  scene  and  to  interact  through  single 
mouse-clicks  if  at  any  point  they  were  “unsure  whether  two  or  more  aircraft  may  safely 
pass  over  or  under  one  another”.  This  very  specific  phrase  was  chosen  to  preclude  any 
inference  of  particular  standard  separation  minima  as  each  controller  separates  aircraft 
differently  at  with  different  timing;  each  controller’s  comfort  level  with  aircraft  proximity 
varies  greatly  upon  the  situation  at  hand  and  innumerable  cognitive  factors  that  are 
ultimately  based  solely  on  the  controller’s  judgment. 

Judgment  is  derived  from  training,  experience  and  techniques  developed  thorough 
firsthand  experience.  Without  a  sophisticated  manner  in  which  to  determine  this 
judgment,  or  awareness,  as  SA  is  a  inherently  difficult  thing  to  measure  (Endsley,  2000), 
this  simple  phrase  was  employed  very  successfully  in  clarifying  to  each  participant 
exactly  what  the  single  mouse-click  was  for  and  when  to  use  it. 

Therefore,  when  examining  a  controller’s  performance  during  use  of  a  new 
display,  their  SA  must  be  considered  both  as  to  support  it  and  also  avoid  hindering  it. 
According  to  Endsley,  “there  is  no  current  agreed-upon  method  for  measuring  SA”, 
(1995).  This  makes  questioning  experiment  participants  post-treatment  the  most 
effective  method  for  gaining  an  understanding  of  the  operator’s  awareness  (Adams, 
Tenney,  and  Pew,  1995). 

Determining  SA  from  query  is  best  done  by  freezing  a  scenario  and  questioning 

the  operator  about  what  they  saw.  However,  this  does  not  actually  capture  their  SA  from 

that  specific  time  which  in  itself  may  have  been  less  clear  and  merely  the  time  elapsed 
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from  the  moment  of  observation  until  the  time  asked,  the  reasons,  activities  or  other 
characteristics  may  have  become  clear  permitting  a  more  acceptable  answer  after  the  fact. 
Contrary  to  this,  it  has  been  shown  that  after  the  fact  recall  tends  to  contain  generalized, 
over-summarized  and  simplified  content  that  may  not  be  an  accurate  assessment  of 
situation  awareness  from  the  previous  exposure,  albeit  immediately  following,  the  real¬ 
time  event  in  question  (Endlsey,  2000;  Nisbett  &  Wilson,  1977). 

There  are  many  measurement  methods  provided  certain  capabilities  are  present, 
unfortunately  for  the  researcher  the  aforementioned  method  of  pausing  for  questions 
would  not  be  feasible  in  a  dynamic  or  live-training  situation  making  it  prohibitive  in  the 
interest  of  flight  safety  as  well  as  unrealistic.  The  manner  in  which  awareness  data  was 
collected  during  this  study  will  be  discussed  later  in  this  paper;  however,  both 
quantitative  and  qualitative  measures  were  taken. 

As  briefly  described  earlier,  all  participants  were  first  administered  a  three  minute 
training  scenario  with  a  two-fold  purpose.  The  first  being  to  ensure  they  would  be  able  to 
see  the  stereoscopic  disparity  as  intended  and  second,  to  allow  them  to  familiarize 
themselves  with  the  processes  we  would  be  using  to  conduct  the  experiment;  it  offered  no 
challenge  and  was  used  only  to  familiarize  them  with  the  equipment  and  experimental 
process  and  to  prevent  them  from  being  caught  “cold”  once  actual  performance  measures 
recording  began.  This  was  found  especially  necessary  after  the  pilot  study  revealed  that 
subjects  really  didn’t  know  what  aspect  of  the  complex  actions  of  a  controller  were  being 
focused  on  and  thus  failing  to  provide  measurable  performance  for  the  aspects  being 
evaluated.  After  the  training  scenario,  there  were  brief  questionnaires  administered  to 
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resemble  the  process  to  be  carried  forward  in  the  measured  exposures.  The  data  collected 
by  these  instruments  was  not  retained  or  analyzed. 

After  the  training  scenario,  the  control  and  experimental  scenarios  were  assigned 
to  the  participants  in  a  predetermined  randomized  order.  Since  the  participants  were 
already  volunteers  randomly  assigned  to  participate  in  the  experiment,  the  control  and 
experimental  conditions  were  assigned  in  random  order  to  the  participants  to  support  a 
within  subjects  design  where  each  participant  serves  as  their  own  control  and  experiment 
group.  Participants  with  odd  numbers  had  the  control  condition  first  and  participants 
with  even  numbers  had  the  experiment  condition  first. 

The  experiment  employed  a  within  subjects  design  to  control  for  learning  and 
order  bias.  Every  subject  was  their  own  control  permitting  the  increased  statistical  power 
afforded  by  a  within- subjects  study  (Erlebacher,  1977).  Keren  stated,  “. .  .the  exclusion  of 
individual  differences  results  in  a  higher  degree  of  sensitivity  to  treatment  effects” 

(1992).  Furthermore,  the  exposure  sequence  was  assigned  randomly,  mitigating  the 
potential  for  learning  bias  and  further  strengthening  the  design  for  optimum  management 
of  participant  pool  and  highest  statistical  yield. 

After  each  eight-minute  scenario,  the  display  was  blanked  and  a  questionnaire 

was  administered  to  each  subject  to  evaluate  effectiveness  of  the  technology  and  their 

perceptions  of  the  specific  performance  measures.  These  included  a  screen  capture  with 

the  aircraft  altitudes  blanked.  Participants  were  asked  to  fill  in  the  datatags  for  aircraft 

altitudes  as  accurately  as  possible.  It  also  contained  questions  about  their  perceptions  of 

the  technology,  confidence  in  their  answers  and  a  place  for  unstructured  feedback.  There 

were  no  time  limitations  placed  on  any  questionnaires.  Please  refer  to  Appendices  C 
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through  H  for  actual  examples  of  the  forms,  the  experimental  timeline  with  script  as  well 
as  additional  views  of  the  experiment  setup. 

Measures 

The  hypotheses  were  tested  and  data  collected  through  a  post  treatment 
questionnaire  and  self-reported  perceptions.  The  only  real-time  feedback  during  the 
experiment  was  that  of  the  mouse  click  inputs  measuring  detection  and  SA  as  mentioned 
previously  in  this  chapter.  Table  1  below  indicates  the  data  collection  measures  used. 
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Table  2:  Data  Collection  Measures  for  Each  Hypothesis 


Hypotheses 

Measures 

HI:  SA/Deteetion 

Mouse  Clicks 

H2:  Improve  Recall  Accuracy 

Self-Reported  Post  Treatment  Questionnaire 
Q&A/Fill-in  Blanks  Screen  Capture 

H3:  Reduce  Fatigue 

Self-Reported  Post  Treatment  Questionnaire 
Q&A  and  Likert  Scale 

H4:  Reduce  Workload 

Self-Reported  Post  Treatment  Questionnaire 
Q&A  and  Likert  Scale 

H5:  Reduce  Task  Difficulty 

Self-Reported  Post  Treatment  Questionnaire 
Q&A  and  Likert  Scale 

H6:  Reduce  Distraction 

Self-Reported  Post  Treatment  Questionnaire 
Q&A  and  Likert  Scale 

Summary 

This  chapter  presented  the  methodology  for  procuring  data  correlating  theory  and 
controller’s  efficiency  through  the  use  of  an  experiment.  This  permitted  the  collection 
and  analysis  of  empirical  data  through  observation  of  actual  controllers  performing 
realistic  detection  functions  similar  to  their  standard  trained  methods.  This  afforded  the 
investigator  a  unique  opportunity  to  gain  insight  into  the  potential  impacts  this 
technology  may  have  on  a  number  of  specific  controller  performance  measures. 
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IV.  Analysis  and  Results 


Overview 

The  following  section  describes  the  statistical  analysis  of  the  collected  data.  The 
processing  was  conducted  with  standard  practices  using  the  commercial  SPSS  statistical 
software  package.  A  t-test  was  used  to  determine  statistical  differences  between 
experimental  and  control  display  condition  means  for  each  hypothesis.  Although  the 
primary  interest  was  in  the  differences  between  groups,  since  many  possible  independent 
variables  (TVs)  were  also  collected  via  the  preliminary  questionnaire,  stepwise  regression 
was  also  used  to  determine  influences  of  other  IVs.  These  variables  were  collected  by 
self-disclosure  and  are  as  follows:  gender,  ATC  experience  in  years,  education  level, 
glasses/corrective  lenses  wear,  eye  surgery,  depth  perception  deficiency,  aviation 
experience  as  pilot,  SA/distraction,  multitask  ability,  acceptance  of  new  technology, 
familiarity  with  3D,  preference  of  2D  vs.  3D  when  given  a  choice  for  entertainment,  trust 
of  3D  views  of  depth,  computer  use  as  percent  of  daily  activity,  fatigue  from  3D  viewing, 
and  perception  of  importance  of  the  ATC  job.  These  variables’  meanings  are  discussed 
further  in  the  following  paragraphs. 

These  variables  were  considered  useful  pertaining  to  this  study  as 
indicators  of  performance.  Distraction  was  a  variable  considered  important  if  one  said 
they  were  easily  distracted.  Gender  was  considered  to  analyze  potential  differences 
between  men  and  women,  but  no  significant  influence  was  found.  Experience  in  ATC 
was  considered  as  a  potential  covariate  that  would  be  presumed  to  have  a  large  influence 
on  performance,  however  less  experience  with  the  technology  could  prevent  bias  against 
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new  displays.  Education  was  considered  a  powerful  potential  indicator  as  one  may 
suspect  educated  participants  may  more  readily  recognize  potential  enhancements  and 
improvements  afforded  by  the  new  display. 

Use  of  corrective  lenses,  having  had  corrective  eye  surgery  and  self-disclosed 
depth  perception  deficiencies  were  considered  due  to  the  visually  intense  nature  of  the 
display  technology  being  tested.  Should  an  individual  be  wearing  lenses  that  could 
distort  or  limit  viewing  angles  or  create  glares  could  impact  performance  results.  A 
deficiency  in  depth  perception  could  limit  one’s  ability  to  view  the  scenarios  as  intended 
with  the  correct  disparity — certainly  a  hindrance  that  could  impact  resulting  performance 
and  serve  to  explain  variances  in  performance. 

Aviation  experience  as  a  pilot  was  considered  due  to  the  familiarity  one  may  have 
as  an  advantage  when  viewing  aircraft  altitude  changes  in  a  3D  model  based  upon  aircraft 
performance  characteristics.  Self  distraction  was  considered  in  order  to  identify  the 
participants  that  are  easily  distracted  while  working  ATC.  Individual  ability  to  multitask 
was  considered  to  identify  the  participants  that  have  trouble  managing  multiple  tasks  at 
once  or  observing  many  things  simultaneously,  a  factor  that  would  also  potentially  limit 
one’s  performance  in  the  experimental  treatment. 

Acceptance  and  preference  was  considered  to  evaluate  participants’  resistance  to 
change  or  readiness  to  adapt  to  new  technologies  and  changes  in  their  regular 
workstation.  Their  preferred  method  for  viewing,  whether  2D  or  3D  when  afforded  a 
choice  was  also  collected.  Familiarity  with  the  proposed  technology  was  also  considered 
in  order  to  study  its  influence  on  participants’  performance  as  many  people  had  not  seen  a 


3D  movie  or  modem  3D  television  recently.  Trust  was  considered  to  detect  the  level  of 
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confidence  the  participant  had  in  the  new  medium,  being  that  a  low  level  of  trust  would 
potentially  degrade  the  performance  enhancement  as  the  operator  second-guessed 
displayed  accuracy.  Computer  use  was  considered  in  order  to  measure  time  spend  using 
computer  and  its  influence  on  participants’  performance. 

Results 

HI:  The  use  of  stereoscopic  3D  digital  radar  displays  will  have  a  positive 
effect  on  controller  situation  awareness  and  conflict  detection. 

The  hypothesis  predicted  a  positive  correlation  between  heightened  SA  through 
detection  of  on-screen  conflicts  when  utilizing  a  stereoscopic  display.  The  dependent 
variable  was  number  of  clicks  indicating  situation  awareness  reference  potential  conflicts. 
A  higher  number  of  clicks  indicated  a  low  awareness  of  actual  conflicts.  The 
independent  variable  was  display  condition,  control  was  group  0,  and  experimental  was 
group  1.  A  /-test  was  conducted  with  a  95%  confidence  interval  (Cl)  to  test  for 
significant  difference  between  control  and  experimental  groups,  see  tables  4  &  5  below. 
Once  significance  was  found,  all  the  IVs  collected  from  the  Pre-questionnaire  were 
considered  in  the  stepwise  regression  model  in  order  to  analyze  the  potential  influence  of 
those  factors  on  controllers’  situation  awareness  and  detection. 

The  data  was  assumed  normally  distributed  as  the  t-test  is  considered  robust  with 
respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of  variance  as 
assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test  was 
run  and  discovered  that  the  control  group  averaged  (3.40  +/-  SD),  a  significantly  higher 
number  of  clicks  than  the  experimental  group  (.91  +/-  1.27)  at  (t(68)  =  7.991,  p  =  .000) 
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indicating  a  significant  difference  in  SA  and  detection  performance  between  the  display 
conditions.  Refer  to  the  /-test  and  descriptive  statistics  tables  below,  Tables  3  and  4. 


Table  3:  HI  SA/Detection  t-test  Results 
Independent  Samples  Test 


Levene's 

Test  for 

Equality  of 

Variances 

t-test  for  Equalih 

/  of  Means 

95%  Confidence 

Sig. 

Interval  of  the 

(2- 

Mean 

Std.  Error 

Difference 

F 

Sig. 

t 

df 

tailed) 

Difference 

Difference 

Lower 

Upper 

Equal 

.009 

.926 

7.991 

68 

.000 

2.486 

.311 

1.865 

3.106 

variances 

assumed 

Equal 

7.991 

67.836 

.000 

2.486 

.311 

1.865 

3.106 

variances 

not 

assumed 

Table  4:  HI  SA/Detection  Descriptive  Statistics 


Dependent  Variable:  SA/Detection  (clicks) 


Group 

Mean 

Std. 

Deviation 

N 

0 

3.40 

1.333 

35 

1 

.91 

1.269 

35 

Total 

2.16 

1.799 

70 

After  determining  significance  between  the  display  conditions  using  the  /-test,  a 

stepwise-regression  was  run  to  determine  what  other  variables  influenced  the  DV.  The 

stepwise  regression  model  indicated  a  variance  explained  with  adjusted  R“  of  .613  with 

51 


an  additional  three  IVs  included  in  the  model;  Fatigue,  Multitask,  and  Experience  in 
years  at  (F( 69)  =  28.357,  p  =  .000).  Refer  to  Tables  5  &  6  below  for  the  stepwise 
regression  model  summary  and  coefficients. 


Table  5:  Model  Summary  for  SA/Detection 


Model  Summary:  SA/Detection 

Model 

R 

R 

Square 

Adjusted 

R 

Square 

1 

.696 

.484 

.477 

2 

.753 

.567 

.554 

3 

.777 

.604 

.587 

4 

.797 

.636 

.613 

Table  6:  Stepwise  Regression  Model  Coefficients  for  SA/Detection 


Coefficients:  SA/Detection 

Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

B 

Std. 

Error 

Beta 

t 

Sig. 

4 

(Constant) 

5.263 

.889 

5.919 

.000 

Group 

-2.486 

.267 

-.696 

-9.296 

.000 

Fatigue 

3D 

1.328 

.515 

.208 

2.577 

.012 

Multitask 

-.519 

.195 

-.200 

-2.666 

.010 

Exper. 

(years) 

.044 

.019 

.190 

2.360 

.021 

It  is  presumed  that  the  characteristic  of  fatigue,  as  a  self-disclosed  subjective 
measure,  equates  to  higher  number  of  clicks  for  those  who  indicated  fatigue.  Multitask, 
also  a  self-disclosed  measure,  shows  fewer  clicks  from  those  who  reported  having  a  high 
aptitude  in  multitasking.  The  more  experienced  participants  clicked  more  frequently, 
although  not  as  much  in  the  experimental  treatment  as  in  the  control.  This  may  be 
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attributed  to  their  conditioning  to  be  more  engaged,  vigilant  and  exercise  more  positive 
control.  It  appeared  that  this  heightened  state  resulted  in  an  overly-enthusiastic  reaction 
when  faced  with  a  very  different  and  unfamiliar  display  in  an  attempt  to  not  miss 
anything,  further  resulting  in  very  pre-emptive  detection  of  possible,  yet  unlikely, 
conflicts. 

H2:  The  use  of  stereoscopic  3D  digital  radar  displays  will  have  a  positive 

effect  on  controller  recall  accuracy  of  aircraft  position. 

This  hypothesis  predicted  a  positive  relationship  between  recall  accuracy  and  3D 
stereoscopic  display  use.  The  dependent  variable  was  recall  accuracy  indicating  the 
subjects  were  able  to  accurately  recall  where  the  aircraft  were  in  the  3D  airspace,  a  lower 
result  indicating  poorer  recall  accuracy.  The  independent  variable  was  display  condition; 
control  was  group  0,  and  experimental  was  group  1.  All  the  IVs  collected,  listed  in  Table 
7,  from  the  Pre-questionnaire  were  considered  in  the  model  in  order  to  analyze  the 
potential  influence  of  those  factors  on  controllers’  recall  accuracy.  A  /-test  was 
conducted  with  a  95%  confidence  interval  (Cl)  to  test  for  significant  difference  between 
control  and  experimental  groups.  Once  significance  was  found,  a  stepwise  regression 
analysis  was  performed  to  detect  influence  of  other  independent  variables  on  Recall 
Accuracy. 

The  data  was  assumed  normally  distributed  as  the  t-test  is  considered  robust  with 

respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of  variance  as 

assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test  was 

run  and  discovered  that  the  control  group  averaged  significantly  higher  recall  accuracy  of 
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(4.81  +/-  1.11)  than  the  experimental  group  (2.73  +/-  1.15)  at  (t(68)  =  7.679,  p  =  .000) 


with  a  difference  of  2.08  (95%  Cl,  1.54  to  2.62)  indicating  a  significant  difference  in 
recall  accuracy  of  aircraft  position  between  the  display  conditions,  however  in  the  reverse 
of  the  expected.  Refer  to  Tables  7  &  8  for  the  t-test  results  and  descriptive  statistics. 

The  control  group  showed  a  better  retention  of  the  information  presented  through 
evaluation  of  recall  accuracy  by  way  of  a  post-treatment  questionnaire.  This  hypothesis 
is  therefore  not  supported;  however  the  non-intuitive  finding  indicates  a  potentially 
valuable  insight. 


Table  7:  H2  Descriptive  Statistics 
Descriptive  Statistics 


Dependent  Variable:  Recall  Acc 


Group 

Mean 

Std. 

Deviation 

N 

0 

4.811429 

1.1113774 

35 

1 

2.733333 

1.1525602 

35 

Total 

3.772381 

1 .5357303 

70 
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Table  8:  H2  t-test  Results 
Independent  Samples  Test 


Levene's 
Test  for 
Equality  of 
Variances 


t-test  for  Equality  of  Means 


F 

Sig. 

t 

df 

Sig. 

(2- 

tailed) 

Mean 

Difference 

Std.  Error 

Difference 

95%  Confidence 

Interval  of  the 

Difference 

Lower 

Upper 

Recall 

Equal 

.421 

.519 

7.679 

68 

.000 

2.0780952 

.2706371 

1 .5380474 

2.6181431 

Acc 

variances 

assumed 

Equal 

7.679 

67.910 

.000 

2.0780952 

.2706371 

1 .5380344 

2.6181560 

variances 

not 

assumed 

After  determining  significance  between  the  display  conditions  using  the  t-test,  a 
stepwise-regression  was  run  to  determine  what  other  variables  influenced  the  DV.  The 
stepwise  regression  model  indicated  a  variance  explained  with  adjusted  R“  of  .562  with 
an  additional  two  IVs  included  in  the  model;  depth  perception  deficiency  and 
SA/distraction  at  (F(69)  =  30.489,  p  =  .000).  Refer  to  Tables  9  &  10  below  for  the 
stepwise  regression  model  summary  and  coefficients. 


Table  9:  Model  Summary  for  Recall  Accuracy 


Model  Summary:  Recall  Accuracy 

Model 

R 

Adjusted  R 

R 

Square 

Square 

1 

.681 

.464 

.457 

2 

.741 

.548 

.535 

3 

.762 

.581 

.562 
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Table  10:  Stepwise  Regression  Model  for  Recall  Accuracy 


Coefficients 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

B 

Std. 

Error 

Beta 

t 

Sig. 

3 

(Constant) 

3.389 

.699 

4.849 

.000 

Group 

-2.078 

.243 

-.681 

-8.551 

.000 

Depth 

Perception  Def 

-2.008 

.526 

-.306 

-3.821 

.000 

SA/Distraction 

.364 

.161 

.181 

2.263 

.027 

This  variance  between  display  conditions  may  be  best  explained  when  one 
considers  the  viewing  advantages  of  the  3D  display.  Several  of  the  participants  reported 
having  trouble  recalling  the  exact  location  of  the  highest  or  lowest  aircraft  in  the 
scenarios  after  the  3D  exposure  attributing  their  “lower  priority”  status  to  their  inability 
to  recall.  This  may  suggest  a  lower  awareness,  but  the  researcher  believes  the  contrary. 

A  controller  is  trained  to  not  rely  on  memory  as  much  as  possible  and  to  utilize  visual, 
audio  and  written  cues  to  gather  needed  information  upon  which  to  make  decisions.  In 
fact,  Hopkin  claims  that  forgetting  information  may  be  just  as  important  as  remembering 
it  in  a  dynamic  memory  situation  like  ATC.  Since  the  status  of  each  track  changes  so 
rapidly,  recalling  the  last  or  last  several  altitudes,  (or  other  flight  characteristics),  may 
interfere  with  the  controller’s  ability  to  remember  the  current  or  most  recent  altitude 
(1980).  This  hindrance  explains  a  valuable  aspect  of  this  result  and  justifies  a  perhaps, 
positive  finding  from  an  unsupported  hypothesis.  When  a  presentation  allows  a  controller 
to  rapidly  determine  priorities,  or  “filter  through  the  chaff’,  based  upon  which  aircraft  are 
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more  likely  to  require  separation  instructions  from  those  which  are  clearly  operating  in  a 
safe  proximity,  there  stands  to  be  a  cognitive  benefit. 

As  previously  disclosed  in  “Limitations”,  the  determination  of  SA  is  a  difficult 
one.  By  virtue  of  the  limited  real-time  feedback  collected,  the  post-treatment 
questionnaires  were  the  best  method  to  measure  this.  However,  the  detection  rates  for 
scripted  conflicts  were  far  better  in  the  3D  scenarios  and  although  participants  were  not 
able  to  recall  from  memory  the  precise  altitudes,  they  were  able  to  detect  that  those 
aircraft  were  not  in  need  of  immediate  attention  and  were  therefore  able  to  place  focus  on 
those  aircraft  that  were  in  positions  of  closer  proximity  and  that  may  need  separation 
instructions.  This  seems  to  have  significant  workload  and  awareness  advantages. 

For  the  control  group  having  a  significantly  better  recall  accuracy,  this  can  easily 
be  attributed  to  a  “scan  rate”  or  the  frequency  of  which  a  controller  visually  “visits”  each 
data  tag  on  his  or  her  screen.  As  earlier  explained  in  the  introduction,  a  controller  must 
frequently  revisit  each  aircraft  target  on  their  scope  and  evaluate  the  heading,  speed  and 
altitude  from  which  they  create  a  3D  mental  model  upon  which  they  base  predictions  of 
where  potential  conflicts  may  be  or  will  develop.  When  this  mental  model  is  less  tasking 
to  create,  because  it  is  already  presented  in  such  a  manner  visually,  the  controller  can 
drastically  reduce  the  superfluous  scans  and  more  easily  “zero-in”  on  the  areas  needing 
their  focus.  Further  research  needs  to  be  conducted  on  this  interesting,  non-intuitive 
result. 
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H3:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease 
controller  fatigue. 

The  hypothesis  predicted  perceived  reduction  in  fatigue  reported  by  controllers 
when  utilizing  a  3D  stereoscopic  display.  The  dependent  variable  was  a  reduction  in 
controller  fatigue  and  the  independent  variable  was  display  condition;  control  was  group 
0,  and  experimental  was  group  1.  A  Most  was  conducted  with  a  95%  confidence  interval 
(Cl)  to  test  for  a  significant  difference  between  control  and  experimental  groups. 

The  data  was  assumed  to  be  normally  distributed  as  the  t-test  is  considered 
robust  with  respect  to  the  assumption  of  normality,  and  homogeneity  of  variance  existed 
as  assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test 
was  run  and  it  was  discovered  no  significant  difference  between  the  control  and 
experimental  groups.  Control  reported,  (1.46  +/-  .919)  with  the  experimental  group  (1.37 
+/-  .910)  at  (r(68)  =  .392,  p  =  .696)  with  a  difference  of  .086  (95%  Cl,  -.350  to  .522) 
indicating  no  significant  variance  between  display  conditions.  This  hypothesis  was 
therefore  not  supported.  Refer  to  tables  1 1  &  12  below  for  t-test  results  and  descriptive 
statistics. 
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Table  11:  H3  t-test  Results 


Independent  Samples  Test 


Levene's  Test 

for  Equality  of 

Variances 

t-test  for  Equality  of  Means 

F 

Sig. 

t 

df 

Sig. 

(2- 

tailed) 

Mean 

Difference 

Std.  Error 

Difference 

95% 

Confidence 

Interval  of  the 

Difference 

Lower 

Upper 

Fatigue  Equal 

.139 

.710 

.392 

68 

.696 

.086 

.219 

-.350 

.522 

variances 

assumed 

Equal 

.392 

67.994 

.696 

.086 

.219 

-.350 

.522 

variances  not 

assumed 

Table  12:  H3  Descriptive  Statistics 


Dependent  Variable:  Fatigue 


Group 

Mean 

Std. 

Deviation 

N 

0 

1.46 

.919 

35 

1 

1.37 

.910 

35 

Total 

1.41 

.909 

70 

Worth  noting  is  the  feedback  from  the  participants  on  the  brevity  of  the  exposure 
not  being  long  enough  to  induce  fatigue.  This  seems  to  be  an  understandable  explanation 
when  one  examines  the  industry  standard  of  approximately  2.0  hours  typical  time  in 
position,  (Shorrock,  2007),  making  an  eight-minute  exposure  far  too  short  to  induce  a 
fatiguing  effect. 
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H4:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease 
controller  perceived  workload 

The  hypothesis  predicted  controllers  would  report  a  reduction  in  perceived 
workload  with  use  of  the  stereoscopic  display.  The  dependent  variable  was  controller 
reported  perceived  workload  and  the  independent  variable  was  display  condition;  control 
was  group  0,  and  experimental  was  group  1.  All  the  IVs  collected  from  the  Pre¬ 
questionnaire  were  considered  in  the  model  in  order  to  analyze  the  potential  influence  of 
those  factors  on  controllers’  perceived  workload. 

The  data  was  assumed  normally  distributed  as  the  t-test  is  considered  robust  with 
respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of  variance  as 
assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test 
was  run  and  discovered  that  the  control  group  averaged  significantly  higher  reported 
workload  (4.74  +/-  1.15)  than  the  experimental  group  (2.03  +/-  1.89)  at  (t( 68)  =  8.61 ,  p  = 
.000)  with  a  difference  of  2.714  (95%  Cl,  2.09  to  3.34)  indicating  a  significant  difference 
in  controller  perceived  workload  between  the  groups.  Please  refer  to  Tables  13  &  14 
below  for  t-Test  results  and  descriptive  statistics. 
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Table  13:  H4  t-test  Results 


Independent  Samples  Test 


Levene's 

Test  for 

Equality  of 

Variances 

t-test  for  Equality  of  Means 

Sig. 

(2- 

Mean 

Std.  Error 

95%  Confidence 

Interval  of  the 

Difference 

F 

Sig. 

t 

df 

tailed) 

Difference 

Difference 

Lower 

Upper 

.271 

.604 

8.616 

68 

.000 

2.7142857 

.3150318 

2.0856496 

3.3429219 

8.616 

64.420 

.000 

2.7142857 

.3150318 

2.0850164 

3.3435551 

Table  14:  H4  Descriptive  Statistics 


Group 

Mean 

Std. 

Deviation 

N 

0 

4.742857 

1.1521101 

35 

1 

2.028571 

1 .4649978 

35 

Total 

3.385714 

1.8921277 

70 

After  determining  significance  between  the  display  conditions  using  the  t-test,  a 
stepwise-regression  was  run  to  determine  what  other  variables  influenced  the  DV.  A 
model  generated  using  a  single  additional  covariate,  S A/Distraction  derived  from  self¬ 
disclosure  from  the  pre-questionnaire,  showed  an  adjusted  R2  of  .598  at  (Ft 69)  =  52.236, 
p  =  .000).  The  S A/Distraction  variable  was  derived  from  those  who  indicated  that  they 
do  get  easily  distracted  when  performing  ATC  duties.  Refer  to  Tables  15  &  16  below  for 

the  stepwise  regression  model  summary  and  coefficients. 

61 


Table  15:  H4  Model  Summary  for  Workload  [Perceived] 


Model  Summary:  Workload  [Perceived] 

Model 

R 

Adjusted  R 

R 

Square 

Square 

1 

.722 

.522 

.515 

2 

.781 

.609 

.598 

Table  16:  H4  Stepwise  Regression  Model  for  Workload  [Perceived] 


Coefficients: 

Workload 

Perceived] 

Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Siq. 

B 

Std. 

Error 

Beta 

2  (Constant) 

7.835 

.824 

9.504 

.000 

Group 

-2.714 

.287 

-.722 

-9.460 

.000 

SA/Distraction 

-.731 

.189 

-.296 

-3.870 

.000 

The  control  groups  reported  more  than  double  the  workload  than  the 
experimental  group  even  though  the  workload  and  traffic  level  were  virtually  identical. 
This  finding  shows  potential  for  the  technology  as  a  workload  management  tool  as  well 
as  decision  aid. 


H5:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease 

controller  perception  of  task  difficulty. 

The  hypothesis  predicted  controllers  will  have  a  reduced  perception  of  task 
difficulty  with  use  of  a  stereoscopic  display.  The  dependent  variable  was  controllers’ 
reported  perception  of  task  difficulty  and  the  independent  variable  was  display  condition; 
control  was  group  0,  and  experimental  was  group  1.  All  the  IVs  collected  from  the  Pre- 
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questionnaire  were  considered  in  the  model  in  order  to  analyze  the  potential  influence  of 
those  factors  on  controllers’  perception  of  task  difficulty. 

A  /-test  was  conducted  with  a  95%  confidence  interval  (Cl)  to  test  for  significant 
difference  between  control  and  experimental  groups,  see  tables  18  &  19  below.  Once 
significance  was  found,  a  stepwise  regression  analysis  was  performed  to  detect  influence 
of  other  independent  variables  on  SA/detection. 

The  data  was  assumed  normally  distributed  as  the  Mcst  is  considered  robust  with 
respect  to  the  assumption  of  normality,  and  the  assumption  of  homogeneity  of  variance  as 
assessed  by  Levene’s  Test  for  Equality  of  Variances  was  violated.  An  independent  Mcst 
was  run  and  it  was  found  to  be  (t(53.27 1 )  =  4.894,  p  =  .000)  with  a  difference  of  1.7 
(95%  Cl,  1.0  to  2.40)  indicating  this  hypothesis  was  supported.  Please  refer  to  Tables  17 
&  18  below  for  t-Test  results  and  descriptive  statistics. 
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Table  17:  H5  t-test  Results 


Independent  Samples  Test 


Levene's  Test 

for  Equality  of 

Variances 

t-test  for  Equality  of  Means 

F 

Sig. 

t 

df 

Sig. 

(2- 

tailed) 

Mean 

Difference 

Std.  Error 

Difference 

95% 

Confidence 

Interval  of  the 

Difference 

Lower 

Upper 

Equal 

f  6.554 

.000 

4.894 

68 

.000 

1.7000 

.3474 

1.0069 

2.3931 

variances 

assumed 

Equal 

4.894 

53.271 

.000 

1.7000 

.3474 

1 .0034 

2.3966 

variances  not 

assumed 

Table  18:  H5  Difficulty  Descriptive  Statistics 


Dependent  Variable  :Difficulty 


Group 

Mean 

Std. 

Deviation 

N 

0 

5.314 

1.7950 

35 

1 

3.614 

1.0006 

35 

Total 

4.464 

1 .6775 

70 

After  determining  significance  between  the  groups  using  the  /-test,  a  stepwise- 
regression  was  run  to  determine  what  other  variables  influenced  the  DV.  three  more 
independent  variables,  SA/Distraction,  Aviation  Experience,  and  Computer  Use, 
produced  a  low  adjusted  R2  of  only  .360  at  (F(69)  =  10.695,  p  =  .000).  The 
SA/Distraction  variable  was  derived  from  those  who  indicated  that  they  do  or  do  not  get 
easily  distracted  when  performing  ATC  duties,  Aviation  experience  was  reported  as  those 
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with  pilot  experience,  and  computer  use  was  measured  from  a  self-disclosed  percentage 
of  daily  work  performed  using  a  computer  to  determine  comfort  and  aptitude  with 
computer  technology.  Very  little  variance  between  display  conditions  was  explained 
using  this  model  therefore  no  definitive  conclusion  could  be  drawn.  Refer  to  Tables  19  & 
20  below  for  the  stepwise  regression  model  summary  and  coefficients. 


Table  19:  H5  Model  Summary  for  Difficulty 


Model  Summary:  Difficulty 

Model 

R 

Adjusted  R 

R 

Square 

Square 

1 

.510 

.260 

.250 

2 

.556 

.309 

.288 

3 

.596 

.355 

.326 

4 

.630 

.397 

Table  20:  H5  Difficulty  Stepwise  Regression  Model 


Coefficients:  Difficulty 

Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

B 

Std. 

Error 

Beta 

t 

Sig. 

4 

(Constant) 

7.076 

.997 

7.099 

.000 

Group 

-1.700 

.321 

-.510 

-5.299 

.000 

SA/Distraction 

-.723 

.225 

-.330 

-3.209 

.002 

Aviation  Exp 

2.030 

.743 

.283 

2.732 

.008 

Computer  Use 

.431 

.202 

.217 

2.130 

.037 
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H6:  The  use  of  stereoscopic  3D  digital  radar  displays  will  decrease 
controller  distraction. 

The  hypothesis  predicted  use  of  a  3D  stereoscopic  display  will  reduce  controller 
reported  distraction.  The  dependent  variable  was  self-reported  controller  perceived 
distraction  caused  by  the  display  type  with  a  higher  number  equated  to  increased 
distraction  reported.  The  independent  variable  was  display  condition,  control  was  group 
0,  and  experimental  was  group.  A  Most  was  conducted  with  a  95%  confidence  interval  to 
test  for  significant  difference  between  control  and  experimental  groups,  see  tables  21  & 

22  below. 

The  data  was  assumed  normally  distributed  as  the  t-test  is  considered  robust  with 
respect  to  the  assumption  of  normality,  and  the  assumption  of  homogeneity  of  variance, 
as  assessed  by  Levene’s  Test  for  Equality  of  Variances,  was  violated.  An  independent  t- 
test  was  then  run  and  it  was  found  that  the  experimental  group  mean  (2.33  +/-  1.55)  was 
not  significantly  lower  than  the  control  group  mean  (1.96  +/-  1.33)  at  (/(66.41 6)  =  -.785, 
p  =  .435)  with  a  difference  of  -.271  (95%  Cl,  -.961  to  .419)  indicating  this  hypothesis  was 
not  supported.  Please  refer  to  Tables  22  &  23  below  for  t-Test  results  and  descriptive 
statistics. 
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Table  21:  H6  Distraction  t-test  Results 


Independent  Samples  Test 


Levene's  Test 

for  Equality  of 

Variances 

t-test  for  Equality  of  Means 

F 

Sig. 

t 

df 

Sig. 

12- 

tailed) 

Mean 

Difference 

Std.  Error 

Difference 

95% 

Confidence 

Interval  of  the 

Difference 

Lower 

Upper 

Distraction  Equal 

.976 

.327 

- 

68 

.435 

-.2714 

.3459 

-.9616 

.4187 

variances 

.785 

assumed 

Equal 

- 

66.416 

.435 

-.2714 

.3459 

-.9619 

.4190 

variances  not 

.785 

assumed 

Table  22:  H6  Distraction  Descriptive  Statistics 


Dependent  Variable:  Distraction 


Group 

Mean 

Std. 

Deviation 

N 

0 

1.957 

1.3305 

35 

1 

2.229 

1 .5546 

35 

Total 

2.093 

1 .4429 
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Summary 

The  findings  of  this  experiment  offer  insight  as  to  what  potential  benefits  this 
technology  now  has  in  this  application.  The  results  show  the  potential  to  improve 
contoller  detection  of  potential  conflicts  when  separation  minima  is  in  jeaopardy  through 
heightened  situation  awareness— a  critical  factor  in  controller  performance.  Further  use  of 
this  technology  whilst  managing  typical  and  realistic  ATC  tasks.  There  is  also  evidence 
in  these  results  that  although  recall  is  not  improved,  perhaps  task  prioritization  is, 
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consequently,  a  proposed  critical  component  of  controller  preformance  as  their 
responsibilities  shift  from  that  of  active  control  to  passive  monitoring. 

According  to  antecdotal  data  collected  ,  88.6%  of  controller’s  reported  the  3D 
display  was  useful.  85.7%  reported  the  3D  display  improved  their  SA.  See  summary. 
Table  23  below,  for  hypotheses  summation. 


Table  23:  Hypotheses  Summary 


HI:  Improve  S A/Detection 

Click 

Supported 

H2:  Improve  recall  accuracy 

Self-reported 

Not  Supported* 

H3:  Reduce  fatigue 

Self-reported 

Not  supported 

H4:  Reduce  workload 

Self-reported 

Supported 

H5:  Reduce  difficulty 

Self-reported 

Supported 

H6:  Reduce  distraction 

Self-reported 

Not  supported 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

The  previous  four  chapters  relayed  the  research  problem,  investigative  question, 
the  approach  and  methodology  to  this  study  as  well  as  the  results  and  findings  post¬ 
experiment.  This  chapter  will  review  the  findings  and  how  they  differ  from  the 
predominant  literature  in  the  field  of  3D  stereoscopic  display  use  in  the  field  of  ATC. 
Also  discussed  is  the  potential  for  future  work  to  build  on  this  study,  the  potential  impact 
of  what  has  been  discovered  and  the  implications  to  today’s,  and  possibly  tomorrow’s, 
Air  Force. 

Conclusions  of  Research 

This  was  an  exploratory  study  to  determine  if  further  research  on  stereoscopic  3D 
displays  is  warranted.  The  findings  of  this  study  show  a  very  strong  correlation  to  easing 
controller  workload  and  increasing  effectiveness  through  a  variety  of  characteristics.  The 
discovery  of  controllers’  acceptance  along  with  their  resulting  positive  feedback  and 
marked  performance  improvements  in  the  detection  of  potential  conflicts,  increased 
situation  awareness,  and  perceived  reduction  in  workload  and  difficulty  are  strong 
indicators  of  the  need  for  more  work  in  the  field. 

Significance  of  Research 

The  findings  of  this  study  have  significance  in  the  field  of  human  factors  and 
performance  evaluation  given  future  air  traffic  controllers  will  likely  perform  their  duties 
as  passive  monitors.  These  findings  may  be  employed  currently  when  testing  the 
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potential  for  new  display  and  interface  technologies.  The  results  indicate  the  potential  for 
controllers  to  monitor  more  aircraft  without  loss  of  situation  awareness  due  to  excessive 
workload,  difficulty  or  distraction.  The  results  also  imply  that  controllers,  with  the  help 
of  this  more  intuitive  and  less  mentally  tasking  display,  may  be  better  prepared  to 
maintain  vigilance  over  larger  sectors  for  longer  periods  of  time,  as  implied  by  the  fact 
that  perceived  difficulty  was  reported  much  lower  with  the  experimental  treatment. 

Safety  considerations  appear  to  be  prevalent  in  the  findings.  In  the  interest  of 
fatigue  management,  although  the  controllers  did  not  report  an  increase  or  decrease  in 
fatigue,  the  reported  decrease  in  perceived  difficulty,  workload  and  increase  in  awareness 
and  detection  lead  the  researcher  to  believe  there  is  fatigue  mitigating  potential  with  this 
concept.  The  increased  awareness  and  detection  of  separation  loss  also  indicate  that 
controllers  may  be  better  equipped  to  maintain  safe  separation  of  aircraft  thereby 
permitting  an  increase  in  traffic  safety  in  the  system  as  a  whole. 

Assumptions/Limitations 

It  was  assumed  from  the  beginning  of  this  research  endeavor  that  the  participant 
pool  will  contain  a  varied  mix  of  both  novice  and  well-seasoned  controllers  with  a 
significant  breadth  in  experience.  This  was  considered  a  strength  as  it  allows  for  more 
insightful  examination,  depending  on  statistical  analysis  configurations,  to  determine 
certain  implications  of  the  technology,  its  employment,  perceptions  by,  and  acceptance 
from,  the  users. 

It  was  also  assumed  that  there  was  an  advantage  provided  by  the  use  of  simulation 
software  procured  from  the  FAA  that  permitted  the  creation  of  test  scenarios  that  were  of 
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a  familiar  nature  to  what  the  subjects  used  routinely.  This  also  provided  the  researcher 
the  ability  to  create  and  administer  realistic  scenarios  in  the  way  of  traffic  flow,  potential 
conflicts  and  aircraft  performance  characteristics.  Along  with  the  familiarity,  it  was  also 
considered  that  all  of  the  participants  had  at  least  some  basic  instruction,  if  not  full 
experience,  in  the  art  of  ATC,  albeit  extremely  fundamental  in  the  cases  of  the  newest  of 
students  with  a  mere  60  days  of  training. 

Some  limitations  were  noted  early  on  in  the  design  as  well.  The  availability  of  the 
equipment  that  best  represents  a  true  workstation  in  size  and  complexity  was  cost 
prohibitive;  therefore  a  standard,  commercially  available  23.5”  3D  monitor  with  a  16:9 
aspect  ratio  was  used.  This  was  not  an  optimum  replacement  for  the  standard  20”  by  20” 
displays  controllers  are  accustomed  to,  however  sufficiently  displayed  the  required  view 
for  the  scenarios  used.  Additionally,  the  active  shutter  glasses  were  mandatory,  and 
although  functioned  seamlessly  with  this  experiment,  would  provide  a  hindrance  in 
continued  studies  that  require  one  to  monitor  adjacent  2D  displays  or  retain  any  good 
peripheral  vision  as  the  arms  were  a  substantial  width  blocking  much  peripheral  vision. 

Additionally  the  inability,  at  the  time  of  the  execution,  to  have  dynamic  real-time 
stereoscopic  conversion  meant  all  exposures  were  recordings  and  thereby  limited  the 
controllers  to  a  passive  monitor  role.  This  was  significant  as  most  operators  are  trained 
and  inclined  to  talk  to  the  pilots  and  issue  instructions  to  alter  flight  paths  and  provide  the 
separation  service,  however  placing  current  controllers  in  a  passive  role  assisted  in 
examining  potential  effects  of  this  new  dynamic.  The  only  real-time  interaction  afforded 
was  the  singular  mouse  click  inputs  as  explained  in  Chapter  III. 
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Recommendations  for  Air  Force  Action 


With  further  examination  and  testing  of  this  concept,  it  seems  future  potential  in 
mitigating  a  number  of  detrimental  human  performance  limitations  can  be  attained 
through  the  use  of  this  technology  in  the  ATC  capacity.  The  USAF  has  the  ability  to 
continue  this  valuable  research  testing  and  to  this  end,  potentially  making  our  air  traffic 
controllers  better  equipped  to  handle  increased  volumes  of  aircraft  whether  in  a  wartime 
or  training  environment.  With  the  known  value  of  air  superiority  in  modern  day  conflict  a 
primary  AF  doctrine,  the  ability  to  gain  access  to,  possess,  and  maximize  the  use  of 
airspace  over  the  battlefield,  the  cost  of  failing  to  develop  a  potential  force  multiplying 
technology  may  be  too  great. 

Recommendations  for  Future  Research 

It  is  suggested  that  future  examination  of  this  concept  be  conducted  with  more  in- 
the-field  experimentation  using  realistic  scenarios  and  active,  proficient  controllers.  In 
these  trials  it  is  recommended  that  the  following  be  investigated  for  potential  further 
human  performance  advantages: 

Real-Time  dynamic  scenarios  allowing  controller  inputs  for  pseudo-pilot 
manipulation  of  scenario  for  temporal  quantification  of  responses/inputs 

Use  of  stereo  mirror  displays  that  allow  for  flicker- free  unlimited  viewing 
angles  by  multiple  users,  light  passive  headgear,  all  in  ambient  lighting 

Access  to  live  “slave”  feed  from  active  Digital  Airport  Surveillance  Radar 
(DASR)  to  conduct  real-time  performance  comparison  2D  vs.  3D 

Applications  including: 

o  High  Volume  Undergraduate  Pilot  Training  environments 
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o  Simulated  high-tempo  mock  wartime  scenarios  such  as  Nellis  AFB’s 
annual  “Red  Flag”  wartime  training  exercise 

o  Advanced  Combat  Control  Training  simulating  wartime  airstrike 
command  and  control 

o  Remotely  Piloted  Aircraft  en  route  formation  cell  control,  break-up 
and  re-join  operations 


Summary 

There  appear  to  be  potential  benefits  to  the  use  of  stereoscopic  3D  radar  displays 
in  today’s  air  traffic  control  environment.  The  research  contained  in  this  paper  is  limited 
by  time,  funding  and  manpower,  however  there  is  more  to  be  explored  and  learned  in  this 
application  with  favorable  results  in  the  field  of  human  performance  and  human  factors 
considerations  for  ATC. 
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Appendix  A 


DEPARTMENT  OF  THE  AIR  FORCE 


HEADQUARTERS  UNITED  STATES  AIR  FORCE 
WASHINGTON  DC 


7  Jan  2012 


MEMORANDUM  FOR  AFIT/ENV 

FROM:  HQ  HAF/A30-BA 

1480  Air  Force  Pentagon 
Washington,  D.C.20330-1480 

SUBJECT:  Accessing  Volunteer  ATC  Trainees  at  Keesler  AFB,  MS  for  AFIT  Research 

1)  TSgt  Russi  and  his  research  team  are  granted  access  to  the  USAF  Air  Traffic  Control 
technical  training  school  to  conduct  their  research  experiment  with  cooperation  of  the  school's 
commandant,  the  career  field  manager  and  any  volunteers  willing  to  participate.  It  is  understood 
that  the  research  project  is  examining  the  potential  of  stereoscopic  (3-D)  digital  radar  display 
potential  and  textual  cueing  in  congested  communications  environments.  Our  tech  school 
provides  the  perfect  environment  for  this  study  without  affecting  operational  ATC  while  also 
allowing  many  AD  controllers  and  apprentices  the  opportunity  to  participate. 

2)  This  support  is  for  the  solicitation  of  volunteers  at  the  school  and  is  not  an  endorsement  or 
expression  of  interest  in  the  specific  technologies  being  evaluated.  We,  as  a  highly  technical 
field,  understand  the  importance  of  research  of  technologies  with  potential  for  future  use  and 
enthusiastically  support  the  AFIT  mission  of  AF  level  research  and  development. 

3)  Please  direct  any  questions  regarding  this  correspondence  to  me  at:  ioseph.kirkfflius. af.mil. 


KIRKJOSEPH  C  102 

I  l.v_.  IU4  Dtnr=US.(WJ'>  Owiin»m.ou=CloC>ou=P* 


DN  r=US.o=tl  $  Gc»«n»m#nT.OM=CoC>  ou=PKt 


8631681 


ou=U  SA  t.  ctertIK  JOSfPM  C  10*6J16*1 
DEM  M1J0207  1646 J905W 


JOSEPH  C.  KIRK,  CMSgt,  USAF 
1C1  Career  Field  Manager 
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Appendix  B 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  BASE  OHIO 


25  Jan  2012 

MEMORANDUM  FOR  LTCOL  BRENT  LANGHALS 

FROM:  Alan  R.  Heminger,  Ph.D. 

AFIT  IRB  Research  Reviewer 
2950  Hobson  Way 

Wright-Patterson  AFB,  OH  45433-7765 

SUBJECT:  Approval  for  exemption  request  from  human  experimentation  requirements  (32 
CFR  219,  DoDD  3216.2  and  AFI  40-402)  for  study  titled  "The  impact  on  performance  of 
stereoscopic  dimensional  digital  radar  displays  and  integrated  automated  visual  text  cuing 
on  air  traffic  controllers". 

1.  Your  request  was  based  on  the  Code  of  Federal  Regulations,  title  32,  part  219,  section  101, 
paragraph  (b)  (2)  (i)  Information  obtained  is  not  recorded  in  such  a  manner  that  human  subjects 
can  be  identified,  directly  or  through  identifiers  linked  to  the  subjects,  (ii)  you  are  not  collecting 
information  that  could  reasonably  place  the  subjects  at  risk  of  criminal  or  civil  liability  or  be 
damaging  to  the  subjects  financial  standing,  employability,  or  reputation,  and  (4)  Research, 
involves  the  collection  or  study  of  existing  data,  documents,  records,  pathological  specimens,  or 
diagnostic  specimens,  if  these  sources  are  publicly  available  or  if  the  information  is  recorded  by 
the  investigator  in  such  a  manner  that  subjects  cannot  be  identified,  directly  or  through 
identifiers  linked  to  the  subjects. 

2.  Your  study  qualifies  for  this  exemption  because  you  are  not  collecting  sensitive  data,  which 
could  reasonably  damage  the  subjects’  financial  standing,  employability,  or  reputation.  Further, 
the  demographic  data  you  are  collecting  will  be  maintained  separately  from  experimental  data  so 
that  a  given  response  will  not  be  expected  to  map  to  a  specific  subject. 

3.  This  determination  pertains  only  to  the  Federal,  Department  of  Defense,  and  Air  Force 
regulations  that  govern  the  use  of  human  subjects  in  research.  Further,  if  a  subject’s  future 
response  reasonably  places  them  at  risk  of  criminal  or  civil  liability  or  is  damaging  to  their 
financial  standing,  employability,  or  reputation,  you  are  required  to  file  an  adverse  event  report 
with  this  office  immediately. 


ALAN  HEMINGER,  PH  D. 
AFIT  Research  Reviewer 


CC.  Laurienne  C.R.A.  Santana,  1  ft,  Brazilian  AF 
Co-investigator 
Jason  G.  Russi 

Co-investigator 
Lori  Kinder,  Contractor 

AFIT  Sponsored  Programs  Office 
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Appendix  C 


Experiment  Scenes 


Figure  11:  Initial  prototype  without  supplementary  data  block  information 


Figure  12:  Illustration  of  double-vision  with  unaided  eye  due  to  stereoscopic  display 
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Figure  13:  Experiment  Setup  During  Data  Collection 


Figure  14:  Experimental  Workstation  Layout 
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Appendix  D 


Experiment  Design  Timeline  (Script) 

TIME 

T  +0  Subjects  arrive  at  experiment  room  1,  adjacent  to  room  2  where  researcher  2  will  initiate 

scenario.  Consent  form  introduced  and  signed  by  subject  (3  min) 

WELCOME: 

“I  am  TSgt  Jason  Russi  and  this  is  my  colleague,  Lt.  Santana  of  the  Brazilian  Air  Force. 
We  would  like  to  welcome  you  and  thank  you  for  your  voluntary  participation  in  this  valuable  research. 
This  study  will  take  approximately  1  hour  and  20  minutes  to  complete.  Upon  completion  of  the  study  you 
will  be  given  a  thank-you  letter  recognizing  your  involvement  in  this  worthwhile  experiment.  Before  we 
begin,  please  take  a  moment  to  read  through  these  consent  forms  Lt.  Santana  is  handing  you  now  and  note 
your  participant  number  written  on  top;  remember  this  number  as  it  will  be  used  as  your  sole  identification 
throughout  the  experiment.  If  you  have  any  questions,  please  do  not  hesitate  to  ask  either  one  of  us  at  any 
point  now  or  during  the  briefings.  Of  note,  your  responses  to  this  experimental  scenario  will  not  be  used 
beyond  our  research  and  will  be  destroyed  upon  completion  of  this  study.  Once  you  have  read  through  the 
entire  consent  form  and  agree  to  continue  to  participate  in  this  study,  please  sign  and  date  the  consent  forms 
and  hand  them  to  either  one  of  us.” 

Wait  for  participant  to  fill  out  consent  form.  (3  min) 

T  +3  Subjects  read  and  fill  in  pre-experiment  questionnaire.  See  Appendix  X  for  full  text  of 

Pre-Experiment  Questionnaire  (PreQ).  Questionairre  will  include  text  and  dimensional 
display  questions  (5min) 

“As  part  of  this  study  we  need  to  collect  some  demographic  and  background  information. 
None  of  this  data  will  be  personally  identifying  and  will  in  no  way  be  connected  to  your  name.  Please  take 
a  moment  to  fill  out  this  survey.” 

Wait  for  participant  to  fill  out  PreQ..  (5  min) 


T+  8  Subject  is  ushered  into  controlled  scenario  room/section  of  area.  Researcher  #1  reads 

script  of  process  and  briefs  subject  on  the  controls  and  familiarizes  them  with  system  that 
will  remain  the  same  throughout  all  three  scenarios.  This  will  constitute  the  training  for 
the  subject’s  interaction  with  system.  Any  questions  are  answered  to  ensure 
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understanding  of  performance  for  subject.  All  scenarios  will  be  briefed,  as  in  what  the 


three  different  types  consist  of  and  inform  the  subject  that  the  order  of  presentation  will 
be  random,  however,  operation  of  the  system  will  be  the  same.  (10  mins) 

TRAINING  BRIEFING 


“For  this  study  you  are  performing  duties  as  an  n  USAF  Air  Traffic  Controller  at  Canyon  Airport, 
and  imaginary  airfield  of  our  design.  You  will  be  tasked  with  controlling  several  aircraft  in  three  separate 
ATC  scenarios.  You  will  be  expected  to  maintain  minimum  standard  vertical  and  lateral  separation 
between  all  aircraft  “tracks”.  This  performance  will  be  evaluated  on  a  series  of  measurable  performance 
characteristics  similar  to  the  standard  accepted  ATC  practices.  You  will  be  expected  to  perform  the 
functions  in  these  simplified  scenarios  allowing  us  to  evaluate  your  decision  making,  timeliness,  conflict 
recognition,  resolution  actions. 

You  have  been  invited  to  participate  in  this  study  due  to  your  background  in  ATC  and  you  are 
expected  to  utilize  any  skills  or  accepted  separation  practices  as  you  would  in  a  live  ATC  setting 
performing  to  the  best  of  your  individual  ability. 

If  you  are  a  student  at  Keesler  AFB,  ATC  Technical  School,  this  will  in  no  way  impact  your 
training  records  or  even  be  shared  with  the  faculty  of  this  school.  There  will  be  no  adverse  actions  or 
retribution  of  any  type  due  to  your  performance.  Furthermore,  there  will  be  absolutely  no  personally 
identifying  information  (PII)  collected  or  retained  on  you  as  an  individual.  Your  participant  number  will  be 
the  only  identifying  information  recorded  for  purposes  of  this  research.” 

Throughout  the  study  you  will  be  monitored  and  timed  as  you  handle  conflicting  aircraft  tracks. 
The  purpose  of  this  study  is  to  determine  the  impact  and  effectiveness  of  the  technologies,  NOT  the 
controllers’  individual  performances  or  abilities.  This  is  a  human-computer  interaction  (HCI)  study  and 
therefore  your  responses  will  be  compared  to  your  own  responses  from  the  baseline  of  the  three  scenarios 
administered  and  not  against  others.” 

TRAINING  BRIEFING  ON  MOCK  CONTROL  TERMINAL  (MCT) 

(**pendinig  development  as  design  evolves) 

-  Conflict  detection  notification  by  subject  to  be  recorded.  A  mouse  click  will 

represent  acknowledgment  of  a  potential  conflict.  This  may  be  followed  with  an  option 

box  posing  the  question  of  how  the  subject  would  like  to  handle  the  conflict,  by  turning, 

by  altitude  separation  or  no  action  and  one  of  the  three  must  be  selected. 


79 


T  +18 


-  To  represent  the  intent  to  communicate  with  an  aircraft,  the  space  bar  will  act 
as  a  mock  Push  to  Talk  (PTT).  This  will  allow  the  measurement  of  the  attempts  to 
communicate  with  the  “pilots”. 

-  Mock  Control  Terminal  (MCT)  orientation 

—  headset  with  mic 

—keyboard,  mouse,  and  any  peripherals  usage 

—  speaker  and  communication  equipment  (if  applicable) 

—  actions  (inputs)  for  making  responses;  I.e.  F-keys  functions 

—  Brief  that  all  three  variations  of  the  scenarios  will  be  15  minute  in 
duration  each  and  that  the  subject  shall  be  focusing  on  conflict 
detection  and  resolution  in  all  despite  varied  appearances. 

Researcher  #1  and  #2  fall  back  to  operator’s  station.)  This  may  be  in  the  same  room, 
depending  upon  space  availability. )  The  first  (control)  scenario  is  initiated  and  recording 
of  measurable  begins.  ALL  SCENARIOS  WILL  BE  ADMINSTERED  IN  A  RANDOM 
MANNER,  this  will  preclude  any  order  effects  that  may  skew  data  collection  by  bias. 
(15mins) 

MEASURABLES  (see  appendix  for  complete  list) 

-  Binary  measurable  =  Yes/No 

1 .  )  Conflict  detection  -  mouse  click,  subsequent  question 

2. )  Conflict  resolution  attempt  -  question  box 

-  Response  Parameters/Quantification  =  Timing,  quantities 

1 . )  Time  to  Detection/recognition  of  imminent  aircraft  conflict 

2. )  Time  to  Resolution  Attempt/decision  made 

3.)  Time  to  initiate  conflict  resolution  will  be  measured  by 
initiation  of  control  instructions  to  de-conflict  tracks.  This  will  allow 
researchers  to  record  time  of  recognition  and  action  taken 
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T  +33 

T  +35 

T  +38 
T  +53 

T  +55 

T  +58 
T  +73 


T+81 


Subject  scenario  terminated.  Subject  briefed  to  wait  in  designated  holding  area  for  next 
scenario  preparation.  (2  min) 

Subject  is  ushered  into  MCT  again.  Researcher  #1  will  ask  subject  if  there  are  any 
additional  questions  about  the  experiment  at  this  time.  (3  mins) 

Begin  next  randomly  selected  15  minute  scenario.  (15  min) 

Subject  scenario  terminated.  Subject  briefed  to  wait  in  designated  holding  area  for  next 
scenario  preparation.  (2  min) 

Subject  is  ushered  into  MCT  again.  Researcher  #1  will  ask  subject  if  there  are  any 
additional  questions  about  the  experiment  at  this  time.  (3  mins) 

Begin  last  remaining  15  minute  scenario  .  (15  mins) 

Subject  is  notified  of  experiment  termination  and  is  ushered  into  holding  area  by 
Researcher  #1  and  is  administered  the  Post-Experiment  Questionairre  (PocQ).  (8  mins) 
***One  Subject  Evaluated  at  T  +81*** 

Simultaneously  Researcher  #2  resets  the  MCT  for  the  next  subject  (3  mins)  then  will  go 
administer  the  Pre -Experiment  Questionairre  (PreQ).  (5  mins) 

(STEP  1,  Subject  #2)  Subject  is  ushered  into  controlled  scenario  room/section  of  area. 
Researcher  #1  reads  script  of  process  and  briefs  subject  on  the  controls  and  familiarizes 
them  with  system  that  will  remain  the  same  throughout  all  three  scenarios.  This  will 
constitute  the  training  for  the  subject’s  interaction  with  system.  Any  questions  are 
answered  to  ensure  understanding  of  performance  for  subject.  All  scenarios  will  be 
briefed,  as  in  what  the  three  different  types  consist  of  and  inform  the  subject  that  the 
order  of  presentation  will  be  random,  however,  operation  of  the  system  will  be  the  same. 
(10  mins) 
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Appendix  E 


Informed  Consent  Document 
For 

The  effect  of  stereoscopic  dimensional  digital  radar  displays  and  integrated  automated  visual 
text  cuing  on  air  traffic  controller  performance 

AFIT/ENV,  Air  Force  Institute  of  Technology,  Wright -Patterson  AFB  OH 


Principal  Investigator:  Lt  Col  Brent  T.  Langhals,  DSN  785-3636,  ext.  4352,  AFIT/ENV 

Brent.  Langhals  @  afit.edu 


Associate  Investigators:  TSgt  Jason  Russi,  DSN  785-3636,  ext.  4352,  AFIT/ENV 

Jason.Russi@afit.edu 


1  LT  Laurienne  Santana  (Brazilian  Air  Force),  DSN  785-3636,  ext. 
4352,  AFIT/ENV 

Laurienne.  S  antana.br  @  afit.  edu 


1.  Nature  and  purpose:  You  are  being  invited  to  take  part  in  a  research  study.  The  information  in  this 
form  is  provided  to  help  you  decide  whether  or  not  to  take  part.  Study  personnel  will  be  available  to 
answer  your  questions  and  provide  additional  information.  If  you  decide  to  take  part  in  the  study,  you 
will  be  asked  to  sign  this  consent  form.  A  copy  of  this  form  will  be  given  to  you.  Your  participation 
will  occur  at  Keesler  AFB,  MS. 


The  purpose  of  this  study  is  to  determine  to  what  extent  visual  textual  cuing  and 
3Dimensional  air  traffic  representations  may  impact  a  controller’s  performance  both  in  heightened 
situational  awareness  for  longer  durations  and  reduction  in  fatigue  induced  by  multi-tasking, 
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distractions,  redundant  transmissions  and  reduced  mental  image  creation.  The  intent  is  to  study  what  of 
these  aspects  can  provide  benefits  to  the  already  task  saturated  industry  by  providing  increased  human 
factors  awareness. 

The  time  requirement  for  each  volunteer  subject  is  anticipated  to  be  a  total  of  one  hour  over 
one  or  two  visits  as  it  need  not  be  continuous.  It  is  expected  approximately  40  subjects  will  be  enrolled 
in  this  study.  Subjects  shall  be  able  to  read  and  speak  English,  be  between  18  and  60  years  old, 
possess  ATC  experience  or  at  least  conceptual  understanding  and  training  with  sufficient  vision  to 
perform  simulated  ATC  tasks  with  a  stereoscopic  monitor. 

2.  Experimental  procedures:  If  you  decide  to  participate,  the  first  task  you  will  complete  will  be  to  fill 
out  a  short  questionnaire  to  capture  some  demographic  information.  No  personally  identifying 
information  will  be  asked  of  you  in  this  questionnaire. 

Next  we  will  ask  you  to  assume  the  role  of  an  air  traffic  controller.  You  will  be  first  presented 
with  a  common  scenario  to  start  the  air  traffic  control  activities,  than  you  will  be  presented  with  a 
series  of  simulated  air  traffic  situations  with  textual  cues,  and  after  that  you  will  be  exposed  to  3D 
displays.  Your  task  is  to  observe  each  simulation  as  different  cues  and  presentations  are  added.  You 
will  be  trained  on  3D  displays  and  have  an  overview  on  the  station  you  will  deal  with  prior  to  the  start 
of  the  experiment.  No  personally  identifying  information  will  be  kept.  About  one  hour  will  be  needed 
to  complete  this  study. 

Your  participation  in  this  study  is  voluntary.  You  will  not  lose  any  benefit  that  would 
normally  be  entitled  if  you  do  not  participate  or  withdraw  from  the  research.  You  may  decide  to  not 
begin  or  to  stop  the  study  at  any  time.  If  you  are  a  student,  your  refusal  to  participate  will  have  no 
effect  on  your  student  status.  Also,  any  new  information  discovered  about  the  research  will  be 
provided  to  you.  This  information  could  affect  your  willingness  to  continue  your  participation  and  will 
therefore  be  furnished  to  you. 
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3. 


Discomfort  and  risks:  The  tasks  that  you  will  be  doing  have  no  known  safety  or  psychological  risks. 
Although  we  have  tried  to  avoid  risks,  you  may  feel  that  some  questions  or  procedures  we  ask  you  to 
do  may  be  stressful,  or  possibly  even  cause  you  to  feel  fatigue.  If  this  occurs  you  can  stop 
participating  immediately,  however  this  is  what  we  are  evaluating  so  please  feel  free  to  express  these 
feelings  and  concerns  as  they  arise.  We  can  give  you  information  about  individuals  who  may  be  able 
to  help  you  with  these  problems  should  they  go  beyond  the  scope  of  the  trial. 


Additionally,  for  this  research  study  we  may  be  using  3D  glasses  for  viewing  the  displays. 
There  are  no  known  risks  from  using  this  equipment  as  they  are  commonly  available  commercial 
models. 


4.  Precautions  for  female  subjects  or  subjects  who  are  or  may  become  pregnant  during  the  course 
of  this  study:  There  are  no  precautions  for  female  subjects  or  subjects  who  are  or  may  become 
pregnant  during  the  course  of  this  study. 

5.  Benefits:  You  are  not  expected  to  benefit  directly  from  participation  in  this  research  study. 

6.  Compensation:  If  you  are  active  duty  military  you  will  receive  your  normal  active  duty  pay. 

7.  Alternatives:  Your  alternative  is  to  choose  not  to  participate  in  this  study.  Refusal  to  participate  will 
involve  no  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  You  may  discontinue 
participation  at  any  time  without  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Notify 
any  investigator  of  this  study  to  discontinue. 


Entitlements  and  confidentiality: 


a.  Records  of  your  participation  in  this  study  may  only  be  disclosed  according  to  federal  law, 

including  the  Federal  Privacy  Act,  5  U.S.C.  552a,  and  its  implementing  regulations  and  the  Health 
Insurance  Portability  and  Accountability  Act  (HIPAA),  and  its  implementing  regulations,  when 
applicable,  and  the  Freedom  of  Information  Act,  5  U.S.C.  Sec  522,  and  its  implementing 
regulations  when  applicable.  Your  personal  information  will  be  stored  in  a  locked  cabinet  in  an 
office  that  is  locked  when  not  occupied.  Electronic  files  containing  your  personal  information  will 
be  password  protected  and  stored  only  on  a  secure  server.  It  is  intended  that  the  only  people 
having  access  to  your  information  will  be  the  researchers  named  above,  the  AFRL  Wright  Site 
IRB  or  any  other  IRB  involved  in  the  review  and  approval  of  this  protocol.  When  no  longer 
needed  for  research  purposes  your  information  will  be  destroyed  in  a  secure  manner  (shredding). 
Complete  confidentiality  cannot  be  promised,  in  particular  for  military  personnel,  whose  health  or 
fitness  for  duty  information  may  be  required  to  be  reported  to  appropriate  medical  or  command 
authorities.  If  such  information  is  to  be  reported,  you  will  be  informed  of  what  is  being  reported 
and  the  reason  for  the  report. 
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b.  Your  entitlements  to  medical  and  dental  care  and/or  compensation  in  the  event  of  injury  are 
governed  by  federal  laws  and  regulations,  and  that  if  you  desire  further  information  you  may 
contact  the  base  legal  office  (ASC/JA,  257-6142  for  Wright-Patterson  AFB). 

c.  If  an  unanticipated  event  (medical  misadventure)  occurs  during  your  participation  in  this  study, 
you  will  be  informed.  If  you  are  not  competent  at  the  time  to  understand  the  nature  of  the  event, 
such  information  will  be  brought  to  the  attention  of  your  next  of  kin  or  other  listed  emergency 
contact. 


Next  of  kin  or  emergency  contact  information: 


Name _  Phone# 


d.  The  decision  to  participate  in  this  research  is  completely  voluntary  on  your  part.  No  one  may 
coerce  or  intimidate  you  into  participating  in  this  program.  You  are  participating  because  you 
want  to.  Lt  Col  Brent  T.  Langhals,  or  an  associate,  has  adequately  answered  any  and  all  questions 
you  have  about  this  study,  your  participation,  and  the  procedures  involved.  Lt  Col  Brent  T. 
Langhals  can  be  reached  at  (937)  255-3636  ext  4352.  Lt  Col  Brent  T.  Langhals,  or  an  associate 
will  be  available  to  answer  any  questions  concerning  procedures  throughout  this  study.  If 
significant  new  findings  develop  during  the  course  of  this  research,  which  may  relate  to  your 
decision  to  continue  participation,  you  will  be  informed.  Refusal  to  participate  will  involve  no 
penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  You  may  discontinue  participation 
at  any  time  without  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Notify  one  of 
the  investigators  of  this  study  to  discontinue. 

e.  Personal  Identifiable  Information  to  be  obtained  for  this  study  includes  gender,  ethnicity,  country 
of  citizenship,  age,  and  experience.  Signing  this  document  in  no  way  alters  your  ability  to  obtain 
medical  treatment  that  is  not  part  of  this  study.  Any  Private  Health  Information  obtained  in  the 
course  of  this  study  may  be  used  by  the  investigator  unless  you  revoke  authorization  to  do  so  in 
writing.  If  your  data  is  disclosed  by  the  investigator  to  one  of  the  parties  listed  above,  those 
parties  may  pass  on  your  data  without  further  notification  to  you.  Data  collected  in  the  course  of 
this  study  may  be  withheld  from  you  by  the  investigator  for  the  duration  of  the  study.  If  withheld, 
your  data  will  be  released  at  the  conclusion  of  the  study. 

f.  Your  participation  in  this  study  will  not  be  photographed,  filmed  or  audio/videotaped. 


YOU  ARE  MAKING  A  DECISION  WHETHER  OR  NOT  TO  PARTICIPATE.  YOUR 
SIGNATURE  INDICATES  THAT  YOU  HAVE  DECIDED  TO  PARTICIPATE  HAVING  READ  THE 
INFORMATION  PROVIDED  ABOVE. 


Volunteer  Signature 


Date 
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Volunteer  Name  (printed) 


Advising  Investigator  Signature _ Date 


Investigator  Name  (printed) 


Witness  Signature _  Date 


Witness  Name  (printed) 


Privacy  Act  Statement 


Authority:  We  are  requesting  disclosure  of  personal  information..  Researchers  are  authorized  to 
collect  personal  information  on  research  subjects  under  The  Privacy  Act -5  USC  552a,  10  USC  55,  10 
USC  8013,  32  CFR  219,  45  CFR  Part  46,  and  EO  9397,  November  1943. 

Purpose:  It  is  possible  that  latent  risks  or  injuries  inherent  in  this  experiment  will  not  be 
discovered  until  sometime  in  the  future.  The  purpose  of  collecting  this  information  is  to  aid  researchers  in 
locating  you  at  a  future  date  if  further  disclosures  are  appropriate. 

Routine  Uses:  Information  may  be  furnished  to  Federal,  State  and  local  agencies  for  any  uses 
published  by  the  Air  Force  in  the  Federal  Register,  52  FR  16431,  to  include,  furtherance  of  the  research 
involved  with  this  study  and  to  provide  medical  care. 

Disclosure:  Disclosure  of  the  requested  information  is  voluntary.  No  adverse  action  whatsoever 
will  be  taken  against  you,  and  no  privilege  will  be  denied  you  based  on  the  fact  you  do  not  disclose  this 
information.  However,  your  participation  in  this  study  may  be  impacted  by  a  refusal  to  provide  this 
information. 
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Appendix  F 


3D  Pre-Experiment  Screening  Questionnaire 


1 .  What  is  your  participant  number?  _ 

2.  What  is  your  gender?  _ 

3.  What  is  your  ATC  experience  (yrs/mos.)  _ 

4.  What  is  your  age?  _ 

5.  What  is  your  highest  level  of  education?  _ 

5.  Do  you  wear  contacts  or  glasses?  _ 

If  so,  are  you  wearing  them  now?  _ 

6.  Have  you  had  corrective  eye  surgery?  _ 

7.  On  a  scale  of  1  to  5,  please  answer  whether  or  not  you  agree  with  the  following 
statements  (circle  one  of  the  numbers).  There  is  no  right  or  wrong  answer. 

Disagree  Agree 


A.  I  can  be  distracted  easily  while  working  ATC. 

B.  I  have  trouble  multitasking  with  many  tasks. 

C.  I  often  ask  pilots  to  repeat  themselves  when  stepped  on. 

D.  I  think  that  new  technologies  can  improve  ATC. 

E.  I  am  familiar  with  3D  displays. 

I.  1  convert  2D  displays  in  a  3D  mental  image  when  working. 

J.  If  cost  was  not  an  issue,  I  would  prefer  3D  entertainment. 
L.  I  rely  heavily  on  computer  inputs  for  my  duties. 

N.  I  have  experienced  dizziness  when  viewing  3D  movies. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

YES  NO 
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O.  I  feel  that  ATC  is  an  important  job. 


YES  NO 


Thank-You  for  your  honest  responses  and  participation  in  this  study. 

Appendix  G 

Post-Control  Questionnaire 

Participant  ID  # _  ( Researcher  use  only. _ ) 

1.)  Using  the  below  screen  capture  from  the  scenario,  answer  the  following  questions; 


a. )  Label  the  aircraft  1-5  based  on  their  altitudes  with  1  being  the  closest  to  the  surface  (lowest). 

b. )  Did  Lancerl  1  or  Orca52  appear  to  be  in  conflict  with  any  other  aircraft?  YES  NO 

c. )  Which  appeared  to  be  the  lowest  aircraft? _ 

d. )  How  much  vertical  separation,  in  thousands  of  feet,  was  between  Famel3  and  N172P? 

(  1,000’  2,000’  3,000  5,000’) 

2).  While  monitoring  the  traffic  scene,  I  observed  the  following:  (circle  one  for  each  choice) 
a.).  The  highest  aircraft  in  the  scenario  was,  (  Wages55  or  Conan25) 
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b.)  Merging  target  procedures  should  have  been  employed  f  0  or 

1 

or  2  or 

3  )  times 

3).  During  the  air  traffic  control  simulation:  (please  circle  one) 

No 

Somewhat 

Yes 

A.  1  felt  fatigued  during  the  scenario. 

1 

2 

3 

4 

5 

6 

7 

B.  I  felt  the  task  was  over  simplified. 

1 

2 

3 

4 

5 

6 

7 

C.  I  found  it  difficult  to  maintain  concentration. 

1 

2 

3 

4 

5 

6 

7 

D.  I  was  distracted  by  the  aircraft  with  no  data  tags. 

1 

2 

3 

4 

5 

6 

7 

E.  It  was  difficult  to  determine  altitude  conflicts. 

1 

2 

3 

4 

5 

6 

7 

F.  I  spent  more  time  focused  on  overlapping  targets. 

1 

2 

3 

4 

5 

6 

7 

G.  I  spent  most  time  looking  at  altitude  readouts  in  datatags. 

1 

2 

3 

4 

5 

6 

7 
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Appendix  H 


Post-Experiment  Questionnaire 

Participant  ID  # _  ( Researcher  use  only : _ ) 

1.)  Using  the  below  screen  capture  from  the  last  2  minutes  of  the  scenario,  label  the  aircraft  in  sequence 
1-9  with  1  being  the  closest  to  the  surface  (lowest). 


2.)  While  monitoring  the  traffic  scene,  I  observed  the  following:  (circle  one  for  each  choice) 

A.  Of  the  two,  (  TEXAN26  or  N275GZ  )  was  lower. 

B.  The  highest  aircraft  in  the  scenario  was,  (  EVAC93  SWA722  UNITED44/H  ). 

C.  Merging  target  procedures  should  have  been  employed  (  0  or  1  or  2  or  3  )  times. 

D.  How  much  vertical  separation,  in  thousands  of  feet,  was  there  between  EVAC93  and  SWA722? 

(  1,000’  2,000’  3,000’  5,000’) 

E.  How  much  vertical  separation  in  thousands  of  feet  appeared  to  be  between  TXAC022/H  and 
RAPTR10?  (  1,000’  2,000'  3,000’  5,000') 


3.)  During  the  air  traffic  control  simulation:  (please  circle  one) 

No 

Somewhat 

Yes 

A.  I  felt  fatigued  during  the  scenario.  1  2 

3  4 

5 

6 

7 

B.  I  felt  the  task  was  over  simplified.  1  2 
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3  4 

5 

6 

7 

C.  I  felt  confident  relying  on  the  3D  display. 


6 


7 


1 

D.  I  found  it  difficult  to  maintain  concentration.  1 

E.  The  tasks  seemed  too  difficult.  1 

G.  3D  enhanced  display  improved  [SA].  1 

H.  I  would  like  to  work  with  3D  more.  1  2 

I.  1  found  it  difficult  to  use  the  3D  display.  1 

J.  I  found  this  3D  display  to  be  useful.  1 

K.  I  used  appearance  of  height  over  data  block  info.  1 


L.  3D  display  was  easier  to  determine  altitude  than  data  text.  1 

M.  I  found  the  text  to  be  easier  to  read  even  when  overlapped.  1 

N.  I  feel  the  3D  display  made  scanning  for  conflicts  easier.  1 


2  3 

2  3 

2  3 
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6).  Often  controllers  choose  to  employ  strategies  to  maintain  awareness,  make  decisions  and 
improve  performance.  Did  you  find  the  3D  display  to  be  conducive  to  your  style  of  scanning? 

Yes  No  If  yes,  please  briefly  describe  how  it  impacted  your  style  or  method: 


7).  Please  indicate  whether  you  felt  altitudes  were  made  more  or  less  apparent  with  the  3D  view. 

W  hy? 
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